feat(tts): gapless Web Audio playback engine for Edge TTS with chapter timeline and seek (#4931)
* feat(tts): add PCM speech-bounds detection for sentence audio trimming Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(tts): add WSOLA time-stretch for pitch-preserved playback rate Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(tts): add sentence duration store with per-voice speaking-rate calibration Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(tts): serve edge audio as ArrayBuffer with in-flight fetch dedup Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(tts): add WebAudioPlayer with gapless chunk scheduling and backpressure Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(tts): play edge TTS through gapless Web Audio pipeline Replaces the per-sentence audio element with trimmed, time-stretched buffers scheduled on the shared AudioContext. Marks dispatch at audible time so schedule-ahead cannot run foliate's cursor past the voice; a decode failure or missing audio skips the chunk instead of wedging the session; pause and resume ride context suspend and resume with no iOS rewind hack; the object-URL cache is gone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(tts): add section timeline with measured and estimated sentence durations Includes the foliate-js submodule bump for the getSentences export (fork branch feat/tts-get-sentences; fork PR must merge before this lands so the pinned SHA resolves). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(tts): expose section playback position and sentence-snapped seeking Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(tts): surface playback position and seek in the media session Position state is clamped, never skipped, so the lock-screen scrubber stays live when estimates overshoot; seekto units map per backend (native ms, web seconds). The AudioContext warms up in the tts-speak gesture path before any await, and the silent keep-alive element now runs on all platforms so desktop hardware media keys survive the removal of the per-sentence audio element. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(tts): add seekable chapter progress bar to the TTS panel The scrubber joins the transport cluster with a thin range-xs track and flanking tabular time labels so it cannot be misgrabbed for the chunky rate slider (which persists a global setting). States: reserved disabled slot until the lazy timeline lands, persists across chapter transitions, optimistic thumb with failure toast, monotonic position, tilde-prefixed estimated totals, sentence-event updates under e-ink. The popup grows only when a timeline-capable client is active. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: record deferred TTS listening-engine follow-ups Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: record background TTS decoupling design decisions Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(tts): slim the panel scrubber to a native track with remaining time Match the footer Jump to Location slider (plain native range: thin track, small thumb) instead of the chunky daisyUI pill, show remaining time with a minus prefix on the right, and drop the This chapter caption. Popup height shrinks accordingly. Verified live in Chrome. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: pin foliate-js to merged main with getSentences export Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(tts): catch autoplay rejection from the keep-alive element Running the silent keep-alive on all platforms exposed an un-awaited play() that headless Chromium rejects without a user gesture, failing CI on unhandled rejections while every test passed. The keep-alive is best-effort; the production path is gesture-qualified. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Huang Xin
@@ -0,0 +1,31 @@
|
||||
# TODOS
|
||||
|
||||
Deferred items from the Edge TTS Web Audio plan review (/autoplan, 2026-07-04).
|
||||
Each was explicitly deferred, not forgotten — see the Decision Audit Trail in
|
||||
`.agents/plans/2026-07-03-edge-tts-webaudio.md`.
|
||||
|
||||
## TTS listening engine follow-ups
|
||||
|
||||
- [ ] Cross-section gapless playback: preload and schedule the next section's first
|
||||
sentence so chapter boundaries are as seamless as sentence boundaries. (M)
|
||||
- [ ] Buffer-ahead indicator in the TTS scrubber (show the prefetched region,
|
||||
YouTube-style). (S)
|
||||
- [ ] Lock-screen ±10s seek offsets in addition to prev/next sentence. (S)
|
||||
- [ ] Persist measured sentence durations per book so a reopened chapter starts
|
||||
with an exact timeline instead of estimates. (S)
|
||||
- [ ] Worker offload for decode + WSOLA if device profiling shows main-thread jank
|
||||
on low-end Android. (S)
|
||||
- [ ] Sticky TTSBar scrubber (panel + lock screen only in the first release;
|
||||
recorded as deliberate in decision #24). (S)
|
||||
- [ ] Provider-agnostic voice source hedge: local neural TTS (e.g. Piper/Kokoro
|
||||
WASM) plugging into WebAudioPlayer/SectionTimeline — the engine is designed
|
||||
for this; see "Strategic framing" in the plan. (L)
|
||||
- [ ] Background chapter prefetch (convert timeline estimates to exact durations
|
||||
ahead of playback). (M)
|
||||
- [ ] Background TTS: decouple session ownership from the reader view via an
|
||||
app-level TTSSessionManager so closing the book keeps TTS playing
|
||||
(headless text supply via section.createDocument(), CFI re-anchoring for
|
||||
highlights on reattach, library now-playing pill). Decided matrix: close
|
||||
book = keep playing; reopen same book = seamless reattach (adopt session,
|
||||
redispatchPosition, lazy doc swap at next section); open a DIFFERENT
|
||||
book = TTS stops; explicit stop / sleep timer = stops. (M)
|
||||
@@ -148,6 +148,9 @@ vi.mock('@/services/tts', () => ({
|
||||
getVoices: vi.fn().mockResolvedValue([]),
|
||||
getVoiceId: vi.fn().mockReturnValue(''),
|
||||
redispatchPosition: vi.fn(),
|
||||
ensureTimeline: vi.fn().mockResolvedValue(null),
|
||||
getPlaybackInfo: vi.fn().mockReturnValue(null),
|
||||
seekToTime: vi.fn().mockResolvedValue(undefined),
|
||||
state: 'idle',
|
||||
addEventListener: vi.fn(),
|
||||
removeEventListener: vi.fn(),
|
||||
@@ -155,12 +158,17 @@ vi.mock('@/services/tts', () => ({
|
||||
});
|
||||
ttsControllerInstances.push(this);
|
||||
}),
|
||||
ensureSharedAudioContext: vi.fn().mockResolvedValue(undefined),
|
||||
}));
|
||||
|
||||
vi.mock('@/libs/mediaSession', () => ({
|
||||
TauriMediaSession: class {},
|
||||
}));
|
||||
|
||||
const { mockMediaSessionRef } = vi.hoisted(() => ({
|
||||
mockMediaSessionRef: { current: null as unknown },
|
||||
}));
|
||||
|
||||
vi.mock('@/utils/ssml', () => ({
|
||||
genSSMLRaw: vi.fn((s: string) => `<speak>${s}</speak>`),
|
||||
parseSSMLLang: vi.fn(() => 'en'),
|
||||
@@ -205,7 +213,7 @@ const { mockDeinitMediaSession } = vi.hoisted(() => ({
|
||||
|
||||
vi.mock('@/app/reader/hooks/useTTSMediaSession', () => ({
|
||||
useTTSMediaSession: () => ({
|
||||
mediaSessionRef: { current: null },
|
||||
mediaSessionRef: mockMediaSessionRef,
|
||||
unblockAudio: vi.fn(),
|
||||
releaseUnblockAudio: vi.fn(),
|
||||
initMediaSession: vi.fn().mockResolvedValue(undefined),
|
||||
@@ -479,3 +487,103 @@ describe('useTTSControl handleHighlightMark cross-section navigation', () => {
|
||||
expect(mockView.goTo).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
describe('useTTSControl media-session position and seek', () => {
|
||||
interface FakeWebMediaSession {
|
||||
setActionHandler: ReturnType<typeof vi.fn>;
|
||||
setPositionState: ReturnType<typeof vi.fn>;
|
||||
handlers: Map<string, (details: MediaSessionActionDetails) => void>;
|
||||
metadata: unknown;
|
||||
playbackState: string;
|
||||
}
|
||||
|
||||
const makeFakeMediaSession = (): FakeWebMediaSession => {
|
||||
const handlers = new Map<string, (details: MediaSessionActionDetails) => void>();
|
||||
return {
|
||||
handlers,
|
||||
metadata: null,
|
||||
playbackState: 'none',
|
||||
setActionHandler: vi.fn((action: string, cb: (d: MediaSessionActionDetails) => void) => {
|
||||
handlers.set(action, cb);
|
||||
}),
|
||||
setPositionState: vi.fn(),
|
||||
};
|
||||
};
|
||||
|
||||
type ControllerMock = {
|
||||
ensureTimeline: ReturnType<typeof vi.fn>;
|
||||
getPlaybackInfo: ReturnType<typeof vi.fn>;
|
||||
seekToTime: ReturnType<typeof vi.fn>;
|
||||
addEventListener: ReturnType<typeof vi.fn>;
|
||||
};
|
||||
|
||||
beforeEach(() => {
|
||||
ttsControllerInstances.length = 0;
|
||||
pendingInitResolvers.length = 0;
|
||||
mockMediaSessionRef.current = makeFakeMediaSession();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
mockMediaSessionRef.current = null;
|
||||
cleanup();
|
||||
});
|
||||
|
||||
const startTTS = async () => {
|
||||
render(<Harness />);
|
||||
await act(async () => {
|
||||
const p = eventDispatcher.dispatch('tts-speak', { bookKey: 'book-1' });
|
||||
for (let i = 0; i < 10; i++) await Promise.resolve();
|
||||
while (pendingInitResolvers.length > 0) pendingInitResolvers.shift()!();
|
||||
await p;
|
||||
});
|
||||
return ttsControllerInstances[0] as ControllerMock;
|
||||
};
|
||||
|
||||
it('registers a seekto handler that seeks the controller in seconds', async () => {
|
||||
const controller = await startTTS();
|
||||
const fake = mockMediaSessionRef.current as FakeWebMediaSession;
|
||||
expect(fake.handlers.has('seekto')).toBe(true);
|
||||
await act(async () => {
|
||||
fake.handlers.get('seekto')!({ seekTime: 42 } as MediaSessionActionDetails);
|
||||
for (let i = 0; i < 5; i++) await Promise.resolve();
|
||||
});
|
||||
expect(controller.seekToTime).toHaveBeenCalledWith(42);
|
||||
});
|
||||
|
||||
it('pushes clamped position state on tts-speak-mark', async () => {
|
||||
const controller = await startTTS();
|
||||
controller.getPlaybackInfo.mockReturnValue({
|
||||
position: 20,
|
||||
duration: 10,
|
||||
measuredFraction: 1,
|
||||
});
|
||||
const markListener = controller.addEventListener.mock.calls.find(
|
||||
([type]) => type === 'tts-speak-mark',
|
||||
)?.[1] as (e: Event) => void;
|
||||
expect(markListener).toBeDefined();
|
||||
await act(async () => {
|
||||
markListener(new CustomEvent('tts-speak-mark', { detail: { text: 'hi', name: '0' } }));
|
||||
for (let i = 0; i < 5; i++) await Promise.resolve();
|
||||
});
|
||||
const fake = mockMediaSessionRef.current as FakeWebMediaSession;
|
||||
expect(fake.setPositionState).toHaveBeenCalledWith({
|
||||
duration: 10,
|
||||
position: 10, // clamped to duration, never skipped
|
||||
playbackRate: 1,
|
||||
});
|
||||
});
|
||||
|
||||
it('does not push position state while the timeline is unavailable', async () => {
|
||||
const controller = await startTTS();
|
||||
controller.getPlaybackInfo.mockReturnValue(null);
|
||||
const markListener = controller.addEventListener.mock.calls.find(
|
||||
([type]) => type === 'tts-speak-mark',
|
||||
)?.[1] as (e: Event) => void;
|
||||
await act(async () => {
|
||||
markListener(new CustomEvent('tts-speak-mark', { detail: { text: 'hi', name: '0' } }));
|
||||
for (let i = 0; i < 5; i++) await Promise.resolve();
|
||||
});
|
||||
const fake = mockMediaSessionRef.current as FakeWebMediaSession;
|
||||
expect(fake.setPositionState).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
import { beforeEach, describe, expect, test, vi } from 'vitest';
|
||||
|
||||
const fetchWithAuthMock = vi.fn();
|
||||
|
||||
vi.mock('@/utils/fetch', () => ({
|
||||
fetchWithAuth: (...args: unknown[]) => fetchWithAuthMock(...args),
|
||||
}));
|
||||
|
||||
vi.mock('@/services/environment', () => ({
|
||||
getAPIBaseUrl: () => 'http://api.test',
|
||||
isTauriAppPlatform: () => false,
|
||||
}));
|
||||
|
||||
import {
|
||||
EdgeSpeechTTS,
|
||||
type EdgeTTSPayload,
|
||||
hashTTSPayload,
|
||||
serializeWordBoundaries,
|
||||
WORD_BOUNDARIES_HEADER,
|
||||
} from '@/libs/edgeTTS';
|
||||
|
||||
const makePayload = (text: string): EdgeTTSPayload => ({
|
||||
lang: 'en',
|
||||
text,
|
||||
voice: 'en-US-AriaNeural',
|
||||
rate: 1.0,
|
||||
pitch: 1.0,
|
||||
});
|
||||
|
||||
const makeResponse = () =>
|
||||
new Response(new Uint8Array([1, 2, 3, 4]).buffer, {
|
||||
status: 200,
|
||||
headers: {
|
||||
[WORD_BOUNDARIES_HEADER]: serializeWordBoundaries([
|
||||
{ offset: 1_000_000, duration: 4_000_000, text: 'hello' },
|
||||
]),
|
||||
},
|
||||
});
|
||||
|
||||
describe('hashTTSPayload', () => {
|
||||
test('is stable for equal payload content', () => {
|
||||
expect(hashTTSPayload(makePayload('abc'))).toBe(hashTTSPayload(makePayload('abc')));
|
||||
});
|
||||
|
||||
test('differs when payload fields differ', () => {
|
||||
expect(hashTTSPayload(makePayload('abc'))).not.toBe(hashTTSPayload(makePayload('abd')));
|
||||
expect(hashTTSPayload({ ...makePayload('abc'), pitch: 1.2 })).not.toBe(
|
||||
hashTTSPayload(makePayload('abc')),
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('createAudioData', () => {
|
||||
beforeEach(() => {
|
||||
fetchWithAuthMock.mockReset();
|
||||
fetchWithAuthMock.mockImplementation(async () => makeResponse());
|
||||
});
|
||||
|
||||
test('returns audio bytes and boundaries from the network on first call', async () => {
|
||||
const tts = new EdgeSpeechTTS('https');
|
||||
const { data, boundaries } = await tts.createAudioData(makePayload('first call text'));
|
||||
expect(new Uint8Array(data)).toEqual(new Uint8Array([1, 2, 3, 4]));
|
||||
expect(boundaries).toHaveLength(1);
|
||||
expect(boundaries[0]!.text).toBe('hello');
|
||||
expect(fetchWithAuthMock).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
test('serves the second call from cache with a fresh, non-detached buffer', async () => {
|
||||
const tts = new EdgeSpeechTTS('https');
|
||||
const payload = makePayload('cache hit text');
|
||||
const first = await tts.createAudioData(payload);
|
||||
const second = await tts.createAudioData(payload);
|
||||
expect(fetchWithAuthMock).toHaveBeenCalledTimes(1);
|
||||
// WebKit's decodeAudioData detaches its input; every call must get its
|
||||
// own copy so replay from cache cannot hand out a detached buffer.
|
||||
expect(second.data).not.toBe(first.data);
|
||||
expect(first.data.byteLength).toBe(4);
|
||||
expect(second.data.byteLength).toBe(4);
|
||||
expect(second.boundaries).toHaveLength(1);
|
||||
});
|
||||
|
||||
test('deduplicates concurrent in-flight fetches for the same payload', async () => {
|
||||
let release: (() => void) | undefined;
|
||||
fetchWithAuthMock.mockImplementation(
|
||||
() =>
|
||||
new Promise((resolve) => {
|
||||
release = () => resolve(makeResponse());
|
||||
}),
|
||||
);
|
||||
const tts = new EdgeSpeechTTS('https');
|
||||
const payload = makePayload('concurrent text');
|
||||
const p1 = tts.createAudioData(payload);
|
||||
const p2 = tts.createAudioData(payload);
|
||||
await Promise.resolve();
|
||||
release!();
|
||||
const [r1, r2] = await Promise.all([p1, p2]);
|
||||
expect(fetchWithAuthMock).toHaveBeenCalledTimes(1);
|
||||
expect(new Uint8Array(r1.data)).toEqual(new Uint8Array(r2.data));
|
||||
});
|
||||
|
||||
test('a failed fetch is not cached; the next call retries', async () => {
|
||||
fetchWithAuthMock.mockImplementationOnce(async () => {
|
||||
throw new Error('network down');
|
||||
});
|
||||
const tts = new EdgeSpeechTTS('https');
|
||||
const payload = makePayload('retry after failure');
|
||||
await expect(tts.createAudioData(payload)).rejects.toThrow('network down');
|
||||
const { data } = await tts.createAudioData(payload);
|
||||
expect(data.byteLength).toBe(4);
|
||||
expect(fetchWithAuthMock).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
});
|
||||
@@ -78,7 +78,7 @@ const makeMetadataFrame = (text: string, offset: number, duration: number) =>
|
||||
],
|
||||
});
|
||||
|
||||
describe('EdgeSpeechTTS.createAudio word boundaries (browser WebSocket path)', () => {
|
||||
describe('EdgeSpeechTTS.createAudioData word boundaries (browser WebSocket path)', () => {
|
||||
beforeEach(() => {
|
||||
wsState.instances.length = 0;
|
||||
(URL as unknown as { createObjectURL?: (blob: Blob) => string }).createObjectURL = vi.fn(
|
||||
@@ -97,7 +97,7 @@ describe('EdgeSpeechTTS.createAudio word boundaries (browser WebSocket path)', (
|
||||
pitch: 1.0,
|
||||
};
|
||||
|
||||
const promise = tts.createAudio(payload);
|
||||
const promise = tts.createAudioData(payload);
|
||||
await vi.waitFor(() => expect(wsState.instances.length).toBe(1));
|
||||
const ws = wsState.instances[0]!;
|
||||
// In a browser (jsdom has `window`), the WebSocket constructor must be
|
||||
@@ -110,8 +110,8 @@ describe('EdgeSpeechTTS.createAudio word boundaries (browser WebSocket path)', (
|
||||
ws.emit('message', { data: makeMetadataFrame('brave', 6000000, 4000000) });
|
||||
ws.emit('message', { data: 'Path:turn.end\r\n\r\n' });
|
||||
|
||||
const { url, boundaries } = await promise;
|
||||
expect(url).toBe('blob:mock-object-url');
|
||||
const { data, boundaries } = await promise;
|
||||
expect(new Uint8Array(data)).toEqual(new Uint8Array([1, 2, 3, 4]));
|
||||
expect(boundaries).toEqual([
|
||||
{ offset: 1000000, duration: 4000000, text: 'Hello' },
|
||||
{ offset: 6000000, duration: 4000000, text: 'brave' },
|
||||
@@ -119,9 +119,9 @@ describe('EdgeSpeechTTS.createAudio word boundaries (browser WebSocket path)', (
|
||||
|
||||
// A second call for the same payload is served from the cache: no new
|
||||
// WebSocket connection, same boundaries.
|
||||
const cached = await tts.createAudio(payload);
|
||||
const cached = await tts.createAudioData(payload);
|
||||
expect(wsState.instances.length).toBe(1);
|
||||
expect(cached.url).toBe('blob:mock-object-url');
|
||||
expect(new Uint8Array(cached.data)).toEqual(new Uint8Array([1, 2, 3, 4]));
|
||||
expect(cached.boundaries).toEqual(boundaries);
|
||||
});
|
||||
|
||||
@@ -129,7 +129,7 @@ describe('EdgeSpeechTTS.createAudio word boundaries (browser WebSocket path)', (
|
||||
const { EdgeSpeechTTS } = await import('@/libs/edgeTTS');
|
||||
const tts = new EdgeSpeechTTS('wss');
|
||||
|
||||
const promise = tts.createAudio({
|
||||
const promise = tts.createAudioData({
|
||||
lang: 'en',
|
||||
text: 'No metadata here',
|
||||
voice: 'en-US-AriaNeural',
|
||||
@@ -171,7 +171,7 @@ describe('word-boundary header (de)serialization', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('EdgeSpeechTTS.createAudio over the HTTPS proxy (word boundaries via header)', () => {
|
||||
describe('EdgeSpeechTTS.createAudioData over the HTTPS proxy (word boundaries via header)', () => {
|
||||
beforeEach(() => {
|
||||
httpState.headers = {};
|
||||
httpState.body = new Uint8Array([1, 2, 3]);
|
||||
@@ -191,7 +191,7 @@ describe('EdgeSpeechTTS.createAudio over the HTTPS proxy (word boundaries via he
|
||||
httpState.headers = { [WORD_BOUNDARIES_HEADER]: serializeWordBoundaries(boundaries) };
|
||||
|
||||
const tts = new EdgeSpeechTTS('https');
|
||||
const result = await tts.createAudio({
|
||||
const result = await tts.createAudioData({
|
||||
lang: 'en',
|
||||
text: 'Hello world https',
|
||||
voice: 'en-US-AriaNeural',
|
||||
@@ -204,7 +204,7 @@ describe('EdgeSpeechTTS.createAudio over the HTTPS proxy (word boundaries via he
|
||||
test('returns empty boundaries when the proxy omits the header', async () => {
|
||||
const { EdgeSpeechTTS } = await import('@/libs/edgeTTS');
|
||||
const tts = new EdgeSpeechTTS('https');
|
||||
const result = await tts.createAudio({
|
||||
const result = await tts.createAudioData({
|
||||
lang: 'en',
|
||||
text: 'No header https',
|
||||
voice: 'en-US-AriaNeural',
|
||||
|
||||
@@ -0,0 +1,302 @@
|
||||
import { afterEach, beforeEach, describe, expect, test, vi } from 'vitest';
|
||||
|
||||
import type { TTSMessageEvent } from '@/services/tts/TTSClient';
|
||||
import type { TTSController } from '@/services/tts/TTSController';
|
||||
import { FakeAudioContext } from './tts-fake-audio';
|
||||
|
||||
// Mock control shared with the hoisted module mock.
|
||||
type MockAudioData = {
|
||||
data: ArrayBuffer;
|
||||
boundaries: Array<{ offset: number; duration: number; text: string }>;
|
||||
};
|
||||
let createAudioDataBehavior: (payloadText: string) => Promise<MockAudioData>;
|
||||
let parsedMarks: Array<{ name: string; text: string; language: string }> = [];
|
||||
|
||||
vi.mock('@/libs/edgeTTS', () => {
|
||||
const voices = [{ id: 'en-US-AriaNeural', name: 'Aria', lang: 'en-US' }];
|
||||
return {
|
||||
EdgeSpeechTTS: class MockEdgeSpeechTTS {
|
||||
static voices = voices;
|
||||
create = vi.fn().mockResolvedValue(undefined);
|
||||
createAudioData = vi
|
||||
.fn()
|
||||
.mockImplementation((payload: { text: string }) => createAudioDataBehavior(payload.text));
|
||||
},
|
||||
EDGE_TTS_PROTOCOL: 'wss',
|
||||
};
|
||||
});
|
||||
|
||||
vi.mock('@/utils/ssml', () => ({
|
||||
parseSSMLMarks: vi.fn(() => ({ marks: parsedMarks })),
|
||||
}));
|
||||
|
||||
vi.mock('@/utils/misc', () => ({
|
||||
getUserLocale: vi.fn((lang: string) => (lang === 'en' ? 'en-US' : lang)),
|
||||
}));
|
||||
|
||||
vi.mock('@/services/tts/TTSUtils', () => ({
|
||||
TTSUtils: {
|
||||
getPreferredVoice: vi.fn(() => null),
|
||||
sortVoicesPreferLocaleFunc: () => () => 0,
|
||||
},
|
||||
}));
|
||||
|
||||
const consoleSpy = {
|
||||
warn: vi.spyOn(console, 'warn').mockImplementation(() => {}),
|
||||
log: vi.spyOn(console, 'log').mockImplementation(() => {}),
|
||||
};
|
||||
void consoleSpy;
|
||||
|
||||
// One second of fake audio bytes: the fake decoder maps 1 byte -> 1 sample at
|
||||
// 24kHz, and all-zero samples make findSpeechBounds keep the full range.
|
||||
const audioOf = (seconds: number): MockAudioData => ({
|
||||
data: new ArrayBuffer(Math.round(seconds * 24000)),
|
||||
boundaries: [],
|
||||
});
|
||||
|
||||
const flush = () => new Promise((resolve) => setTimeout(resolve, 0));
|
||||
|
||||
let rafCallbacks: Map<number, FrameRequestCallback>;
|
||||
let rafId = 0;
|
||||
const runRaf = () => {
|
||||
const cbs = [...rafCallbacks.values()];
|
||||
rafCallbacks.clear();
|
||||
for (const cb of cbs) cb(0);
|
||||
};
|
||||
|
||||
interface MockController {
|
||||
dispatchSpeakMark: ReturnType<typeof vi.fn>;
|
||||
prepareSpeakWords: ReturnType<typeof vi.fn>;
|
||||
dispatchSpeakWord: ReturnType<typeof vi.fn>;
|
||||
}
|
||||
|
||||
type EdgeClientClass = typeof import('@/services/tts/EdgeTTSClient').EdgeTTSClient;
|
||||
|
||||
describe('EdgeTTSClient Web Audio playback', () => {
|
||||
let EdgeTTSClient: EdgeClientClass;
|
||||
let controller: MockController;
|
||||
|
||||
beforeEach(async () => {
|
||||
vi.resetModules();
|
||||
FakeAudioContext.instances = [];
|
||||
rafCallbacks = new Map();
|
||||
vi.stubGlobal('AudioContext', FakeAudioContext);
|
||||
vi.stubGlobal('requestAnimationFrame', (cb: FrameRequestCallback) => {
|
||||
rafCallbacks.set(++rafId, cb);
|
||||
return rafId;
|
||||
});
|
||||
vi.stubGlobal('cancelAnimationFrame', (id: number) => {
|
||||
rafCallbacks.delete(id);
|
||||
});
|
||||
createAudioDataBehavior = async () => audioOf(1);
|
||||
parsedMarks = [
|
||||
{ name: '0', text: 'First sentence.', language: 'en' },
|
||||
{ name: '1', text: 'Second sentence.', language: 'en' },
|
||||
];
|
||||
controller = {
|
||||
dispatchSpeakMark: vi.fn(),
|
||||
prepareSpeakWords: vi.fn(),
|
||||
dispatchSpeakWord: vi.fn(),
|
||||
};
|
||||
({ EdgeTTSClient } = await import('@/services/tts/EdgeTTSClient'));
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.unstubAllGlobals();
|
||||
});
|
||||
|
||||
const startClient = async () => {
|
||||
const client = new EdgeTTSClient(controller as unknown as TTSController);
|
||||
await client.init();
|
||||
return client;
|
||||
};
|
||||
|
||||
const collectSpeak = (client: InstanceType<EdgeClientClass>, signal: AbortSignal) => {
|
||||
const events: TTSMessageEvent[] = [];
|
||||
const done = (async () => {
|
||||
for await (const event of client.speak('<ssml/>', signal)) {
|
||||
events.push(event);
|
||||
}
|
||||
})();
|
||||
return { events, done };
|
||||
};
|
||||
|
||||
const ctx = () => FakeAudioContext.instances[0]!;
|
||||
|
||||
test('plays marks gaplessly: boundary per audible chunk, one final end', async () => {
|
||||
const client = await startClient();
|
||||
const { events, done } = collectSpeak(client, new AbortController().signal);
|
||||
await flush();
|
||||
await flush();
|
||||
|
||||
// Both chunks scheduled ahead, but only chunk 0 is audible: exactly one
|
||||
// mark dispatched so foliate's cursor tracks the voice, not the fetcher.
|
||||
expect(ctx().sources.length).toBe(2);
|
||||
expect(controller.dispatchSpeakMark).toHaveBeenCalledTimes(1);
|
||||
expect(events.filter((e) => e.code === 'boundary')).toHaveLength(1);
|
||||
|
||||
await ctx().advanceTo(1.1); // chunk 0 ends (starts at 0.03, 1s long)
|
||||
await flush();
|
||||
expect(controller.dispatchSpeakMark).toHaveBeenCalledTimes(2);
|
||||
|
||||
await ctx().advanceTo(3); // chunk 1 ends
|
||||
await done;
|
||||
expect(events.map((e) => e.code)).toEqual(['boundary', 'boundary', 'end']);
|
||||
expect(events[0]!.mark).toBe('0');
|
||||
expect(events[1]!.mark).toBe('1');
|
||||
});
|
||||
|
||||
test('chunks are scheduled with a rate-scaled gap and no element restarts', async () => {
|
||||
const client = await startClient();
|
||||
await client.setRate(1); // gap = 0.15 / 1
|
||||
const { done } = collectSpeak(client, new AbortController().signal);
|
||||
await flush();
|
||||
await flush();
|
||||
const [first, second] = ctx().sources;
|
||||
expect(second!.startedAt! - first!.endTime).toBeCloseTo(0.15, 5);
|
||||
await ctx().advanceTo(5);
|
||||
await done;
|
||||
});
|
||||
|
||||
test('word tracking follows the audio clock and survives pause/resume', async () => {
|
||||
createAudioDataBehavior = async () => ({
|
||||
data: new ArrayBuffer(48000), // 2s
|
||||
boundaries: [
|
||||
{ offset: 1_000_000, duration: 4_000_000, text: 'Hello' }, // 0.1s
|
||||
{ offset: 6_000_000, duration: 4_000_000, text: 'brave' }, // 0.6s
|
||||
{ offset: 11_000_000, duration: 4_000_000, text: 'world' }, // 1.1s
|
||||
],
|
||||
});
|
||||
parsedMarks = [{ name: '0', text: 'Hello brave world', language: 'en' }];
|
||||
const client = await startClient();
|
||||
const { done } = collectSpeak(client, new AbortController().signal);
|
||||
await flush();
|
||||
await flush();
|
||||
|
||||
expect(controller.prepareSpeakWords).toHaveBeenCalledWith(['Hello', 'brave', 'world']);
|
||||
ctx().currentTime = 0.03 + 0.15;
|
||||
runRaf();
|
||||
expect(controller.dispatchSpeakWord).toHaveBeenLastCalledWith(0);
|
||||
|
||||
await client.pause();
|
||||
expect(ctx().state).toBe('suspended');
|
||||
await client.resume();
|
||||
expect(ctx().state).toBe('running');
|
||||
|
||||
ctx().currentTime = 0.03 + 0.7;
|
||||
runRaf();
|
||||
expect(controller.dispatchSpeakWord).toHaveBeenLastCalledWith(1);
|
||||
|
||||
// Same index is not re-dispatched.
|
||||
const calls = controller.dispatchSpeakWord.mock.calls.length;
|
||||
runRaf();
|
||||
expect(controller.dispatchSpeakWord.mock.calls.length).toBe(calls);
|
||||
|
||||
await ctx().advanceTo(5);
|
||||
await done;
|
||||
});
|
||||
|
||||
test('abort mid-stream yields Aborted and stops all sources', async () => {
|
||||
const client = await startClient();
|
||||
const abortController = new AbortController();
|
||||
const { events, done } = collectSpeak(client, abortController.signal);
|
||||
await flush();
|
||||
await flush();
|
||||
abortController.abort();
|
||||
await done;
|
||||
expect(events.at(-1)).toMatchObject({ code: 'error', message: 'Aborted' });
|
||||
expect(ctx().sources.every((s) => s.stopped)).toBe(true);
|
||||
});
|
||||
|
||||
test('a no-audio mark is skipped and the session continues', async () => {
|
||||
createAudioDataBehavior = async (text: string) => {
|
||||
if (text === 'First sentence.') throw new Error('No audio data received.');
|
||||
return audioOf(1);
|
||||
};
|
||||
const client = await startClient();
|
||||
const { events, done } = collectSpeak(client, new AbortController().signal);
|
||||
await flush();
|
||||
await flush();
|
||||
await ctx().advanceTo(5);
|
||||
await done;
|
||||
const codes = events.map((e) => e.code);
|
||||
expect(codes.filter((c) => c === 'boundary')).toHaveLength(1);
|
||||
expect(codes.at(-1)).toBe('end');
|
||||
});
|
||||
|
||||
test('a decode failure is treated like no-audio: warn, skip, continue', async () => {
|
||||
// The context exists once the scheduler's first fetch runs (ensureContext
|
||||
// precedes it), so the first fetch installs a decoder that fails exactly
|
||||
// once — the first mark's decode dies, the second succeeds.
|
||||
let installed = false;
|
||||
createAudioDataBehavior = async () => {
|
||||
if (!installed) {
|
||||
installed = true;
|
||||
const context = FakeAudioContext.instances[0]!;
|
||||
const original = context.decodeImpl;
|
||||
let failed = false;
|
||||
context.decodeImpl = async (data) => {
|
||||
if (!failed) {
|
||||
failed = true;
|
||||
throw new Error('bad mp3');
|
||||
}
|
||||
return original(data);
|
||||
};
|
||||
}
|
||||
return audioOf(1);
|
||||
};
|
||||
const client = await startClient();
|
||||
const { events, done } = collectSpeak(client, new AbortController().signal);
|
||||
await flush();
|
||||
await flush();
|
||||
await ctx().advanceTo(10);
|
||||
await done;
|
||||
const codes = events.map((e) => e.code);
|
||||
expect(codes.filter((c) => c === 'boundary')).toHaveLength(1); // only mark 1 played
|
||||
expect(codes.at(-1)).toBe('end');
|
||||
});
|
||||
|
||||
test('all marks failing still ends the session with end (no wedge)', async () => {
|
||||
createAudioDataBehavior = async () => {
|
||||
throw new Error('No audio data received.');
|
||||
};
|
||||
const client = await startClient();
|
||||
const { events, done } = collectSpeak(client, new AbortController().signal);
|
||||
await done; // zero chunks scheduled; session-end fires synchronously
|
||||
const codes = events.map((e) => e.code);
|
||||
expect(codes.at(-1)).toBe('end');
|
||||
expect(codes).not.toContain('boundary');
|
||||
});
|
||||
|
||||
test('a hard fetch error yields error and terminates', async () => {
|
||||
createAudioDataBehavior = async () => {
|
||||
throw new Error('network exploded');
|
||||
};
|
||||
const client = await startClient();
|
||||
const { events, done } = collectSpeak(client, new AbortController().signal);
|
||||
await done;
|
||||
expect(events.at(-1)).toMatchObject({ code: 'error', message: 'network exploded' });
|
||||
}, 10000);
|
||||
|
||||
test('pause without a session is a no-op returning true', async () => {
|
||||
const client = await startClient();
|
||||
expect(await client.pause()).toBe(true);
|
||||
expect(await client.resume()).toBe(true);
|
||||
});
|
||||
|
||||
test('getChunkPosition reports trim-relative clamped seconds', async () => {
|
||||
const client = await startClient();
|
||||
parsedMarks = [{ name: '0', text: 'Only sentence.', language: 'en' }];
|
||||
const { done } = collectSpeak(client, new AbortController().signal);
|
||||
await flush();
|
||||
await flush();
|
||||
ctx().currentTime = 0.03 + 0.4;
|
||||
const pos = client.getChunkPosition();
|
||||
expect(pos).not.toBeNull();
|
||||
expect(pos!).toBeGreaterThan(0.3);
|
||||
expect(pos!).toBeLessThanOrEqual(1);
|
||||
await ctx().advanceTo(5);
|
||||
await done;
|
||||
expect(client.getChunkPosition()).toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -3,14 +3,13 @@ import { describe, test, expect, vi, beforeEach, afterEach } from 'vitest';
|
||||
// Shared mock control: tests can override createBehavior to change how create() behaves
|
||||
let createBehavior: () => Promise<undefined> = () => Promise.resolve(undefined);
|
||||
|
||||
// Shared mock control for createAudioUrl() and parsed SSML marks
|
||||
let createAudioUrlBehavior = vi.fn<() => Promise<string>>(() => Promise.resolve('blob:mock-url'));
|
||||
type MockAudioResult = {
|
||||
url: string;
|
||||
// Shared mock control for createAudioData() and parsed SSML marks
|
||||
type MockAudioData = {
|
||||
data: ArrayBuffer;
|
||||
boundaries: Array<{ offset: number; duration: number; text: string }>;
|
||||
};
|
||||
let createAudioBehavior = vi.fn<() => Promise<MockAudioResult>>(() =>
|
||||
Promise.resolve({ url: 'blob:mock-url', boundaries: [] }),
|
||||
let createAudioDataBehavior = vi.fn<() => Promise<MockAudioData>>(() =>
|
||||
Promise.resolve({ data: new ArrayBuffer(8), boundaries: [] }),
|
||||
);
|
||||
let parsedMarks: Array<{ name: string; text: string; language: string }> = [];
|
||||
|
||||
@@ -27,8 +26,7 @@ vi.mock('@/libs/edgeTTS', () => {
|
||||
EdgeSpeechTTS: class MockEdgeSpeechTTS {
|
||||
static voices = voices;
|
||||
create = vi.fn().mockImplementation(() => createBehavior());
|
||||
createAudioUrl = vi.fn().mockImplementation(() => createAudioUrlBehavior());
|
||||
createAudio = vi.fn().mockImplementation(() => createAudioBehavior());
|
||||
createAudioData = vi.fn().mockImplementation(() => createAudioDataBehavior());
|
||||
},
|
||||
EDGE_TTS_PROTOCOL: 'wss',
|
||||
};
|
||||
@@ -70,9 +68,8 @@ describe('EdgeTTSClient', () => {
|
||||
|
||||
beforeEach(() => {
|
||||
createBehavior = () => Promise.resolve(undefined);
|
||||
createAudioUrlBehavior = vi.fn<() => Promise<string>>(() => Promise.resolve('blob:mock-url'));
|
||||
createAudioBehavior = vi.fn<() => Promise<MockAudioResult>>(() =>
|
||||
Promise.resolve({ url: 'blob:mock-url', boundaries: [] }),
|
||||
createAudioDataBehavior = vi.fn<() => Promise<MockAudioData>>(() =>
|
||||
Promise.resolve({ data: new ArrayBuffer(8), boundaries: [] }),
|
||||
);
|
||||
parsedMarks = [];
|
||||
client = new EdgeTTSClient();
|
||||
@@ -431,14 +428,14 @@ describe('EdgeTTSClient', () => {
|
||||
}
|
||||
};
|
||||
|
||||
test('retries createAudioUrl up to 3 times when preload fails', async () => {
|
||||
test('retries createAudioData up to 3 times when preload fails', async () => {
|
||||
await client.init();
|
||||
parsedMarks = [{ name: 'mark-0', text: 'hello', language: 'en' }];
|
||||
createAudioUrlBehavior = vi.fn(() => Promise.reject(new Error('network error')));
|
||||
createAudioDataBehavior = vi.fn(() => Promise.reject(new Error('network error')));
|
||||
|
||||
await consumePreload(client, new AbortController().signal);
|
||||
|
||||
expect(createAudioUrlBehavior).toHaveBeenCalledTimes(3);
|
||||
expect(createAudioDataBehavior).toHaveBeenCalledTimes(3);
|
||||
});
|
||||
|
||||
test('does not retry when the first preload attempt succeeds', async () => {
|
||||
@@ -447,37 +444,47 @@ describe('EdgeTTSClient', () => {
|
||||
|
||||
await consumePreload(client, new AbortController().signal);
|
||||
|
||||
expect(createAudioUrlBehavior).toHaveBeenCalledTimes(1);
|
||||
expect(createAudioDataBehavior).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
test('stops retrying once an attempt succeeds', async () => {
|
||||
await client.init();
|
||||
parsedMarks = [{ name: 'mark-0', text: 'hello', language: 'en' }];
|
||||
let calls = 0;
|
||||
createAudioUrlBehavior = vi.fn(() => {
|
||||
createAudioDataBehavior = vi.fn(() => {
|
||||
calls++;
|
||||
return calls < 2
|
||||
? Promise.reject(new Error('network error'))
|
||||
: Promise.resolve('blob:mock-url');
|
||||
: Promise.resolve({ data: new ArrayBuffer(8), boundaries: [] });
|
||||
});
|
||||
|
||||
await consumePreload(client, new AbortController().signal);
|
||||
|
||||
expect(createAudioUrlBehavior).toHaveBeenCalledTimes(2);
|
||||
expect(createAudioDataBehavior).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
test('does not retry a permanent no-audio failure', async () => {
|
||||
await client.init();
|
||||
parsedMarks = [{ name: 'mark-0', text: 'hello', language: 'en' }];
|
||||
createAudioDataBehavior = vi.fn(() => Promise.reject(new Error('No audio data received.')));
|
||||
|
||||
await consumePreload(client, new AbortController().signal);
|
||||
|
||||
expect(createAudioDataBehavior).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
test('stops retrying once the signal is aborted', async () => {
|
||||
await client.init();
|
||||
parsedMarks = [{ name: 'mark-0', text: 'hello', language: 'en' }];
|
||||
const controller = new AbortController();
|
||||
createAudioUrlBehavior = vi.fn(() => {
|
||||
createAudioDataBehavior = vi.fn(() => {
|
||||
controller.abort();
|
||||
return Promise.reject(new Error('network error'));
|
||||
});
|
||||
|
||||
await consumePreload(client, controller.signal);
|
||||
|
||||
expect(createAudioUrlBehavior).toHaveBeenCalledTimes(1);
|
||||
expect(createAudioDataBehavior).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -496,137 +503,4 @@ describe('EdgeTTSClient', () => {
|
||||
await expect(client.stop()).resolves.toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe('word boundary tracking during playback', () => {
|
||||
class MockAudio {
|
||||
static instances: MockAudio[] = [];
|
||||
src = '';
|
||||
currentTime = 0;
|
||||
preload = '';
|
||||
playbackRate = 1;
|
||||
onended: ((e?: Event) => void) | null = null;
|
||||
onerror: ((e?: unknown) => void) | null = null;
|
||||
constructor() {
|
||||
MockAudio.instances.push(this);
|
||||
}
|
||||
setAttribute() {}
|
||||
play() {
|
||||
return Promise.resolve();
|
||||
}
|
||||
pause() {}
|
||||
}
|
||||
|
||||
let rafCallbacks: Map<number, FrameRequestCallback>;
|
||||
let rafId = 0;
|
||||
const runRaf = () => {
|
||||
const cbs = [...rafCallbacks.values()];
|
||||
rafCallbacks.clear();
|
||||
for (const cb of cbs) cb(0);
|
||||
};
|
||||
const flush = () => new Promise((resolve) => setTimeout(resolve, 0));
|
||||
|
||||
let mockController: {
|
||||
dispatchSpeakMark: ReturnType<typeof vi.fn>;
|
||||
prepareSpeakWords: ReturnType<typeof vi.fn>;
|
||||
dispatchSpeakWord: ReturnType<typeof vi.fn>;
|
||||
};
|
||||
|
||||
beforeEach(() => {
|
||||
MockAudio.instances = [];
|
||||
rafCallbacks = new Map();
|
||||
vi.stubGlobal('Audio', MockAudio);
|
||||
vi.stubGlobal('requestAnimationFrame', (cb: FrameRequestCallback) => {
|
||||
rafCallbacks.set(++rafId, cb);
|
||||
return rafId;
|
||||
});
|
||||
vi.stubGlobal('cancelAnimationFrame', (id: number) => {
|
||||
rafCallbacks.delete(id);
|
||||
});
|
||||
mockController = {
|
||||
dispatchSpeakMark: vi.fn(),
|
||||
prepareSpeakWords: vi.fn(),
|
||||
dispatchSpeakWord: vi.fn(),
|
||||
};
|
||||
client = new EdgeTTSClient(mockController as unknown as TTSController);
|
||||
parsedMarks = [{ name: '0', text: 'Hello brave world', language: 'en' }];
|
||||
createAudioBehavior = vi.fn(() =>
|
||||
Promise.resolve({
|
||||
url: 'blob:mock-url',
|
||||
boundaries: [
|
||||
{ offset: 1_000_000, duration: 4_000_000, text: 'Hello' },
|
||||
{ offset: 6_000_000, duration: 4_000_000, text: 'brave' },
|
||||
{ offset: 11_000_000, duration: 4_000_000, text: 'world' },
|
||||
],
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.unstubAllGlobals();
|
||||
});
|
||||
|
||||
const startSpeak = async () => {
|
||||
await client.init();
|
||||
const it = client.speak('<ssml/>', new AbortController().signal);
|
||||
const first = await it.next();
|
||||
expect((first.value as { code: string }).code).toBe('boundary');
|
||||
const resultPromise = it.next();
|
||||
await flush();
|
||||
const audio = MockAudio.instances.at(-1)!;
|
||||
return { it, resultPromise, audio };
|
||||
};
|
||||
|
||||
test('prepares speak words and dispatches word indexes as playback advances', async () => {
|
||||
const { resultPromise, audio } = await startSpeak();
|
||||
|
||||
expect(mockController.prepareSpeakWords).toHaveBeenCalledWith(['Hello', 'brave', 'world']);
|
||||
|
||||
audio.currentTime = 0.11;
|
||||
runRaf();
|
||||
expect(mockController.dispatchSpeakWord).toHaveBeenCalledWith(0);
|
||||
|
||||
audio.currentTime = 0.65;
|
||||
runRaf();
|
||||
expect(mockController.dispatchSpeakWord).toHaveBeenLastCalledWith(1);
|
||||
|
||||
// Same word index is not re-dispatched on subsequent frames.
|
||||
const callCount = mockController.dispatchSpeakWord.mock.calls.length;
|
||||
runRaf();
|
||||
expect(mockController.dispatchSpeakWord.mock.calls.length).toBe(callCount);
|
||||
|
||||
audio.onended?.();
|
||||
const result = await resultPromise;
|
||||
expect((result.value as { code: string }).code).toBe('end');
|
||||
});
|
||||
|
||||
test('stops dispatching after the chunk ends', async () => {
|
||||
const { resultPromise, audio } = await startSpeak();
|
||||
|
||||
audio.currentTime = 0.11;
|
||||
runRaf();
|
||||
const callCount = mockController.dispatchSpeakWord.mock.calls.length;
|
||||
|
||||
audio.onended?.();
|
||||
await resultPromise;
|
||||
|
||||
audio.currentTime = 1.2;
|
||||
runRaf();
|
||||
expect(mockController.dispatchSpeakWord.mock.calls.length).toBe(callCount);
|
||||
});
|
||||
|
||||
test('hands empty words to the controller and does not track when no boundaries', async () => {
|
||||
createAudioBehavior = vi.fn(() => Promise.resolve({ url: 'blob:mock-url', boundaries: [] }));
|
||||
const { resultPromise, audio } = await startSpeak();
|
||||
|
||||
// Empty words are still forwarded so the controller can draw the
|
||||
// sentence-highlight fallback; no per-word tracking is started.
|
||||
expect(mockController.prepareSpeakWords).toHaveBeenCalledWith([]);
|
||||
audio.currentTime = 0.5;
|
||||
runRaf();
|
||||
expect(mockController.dispatchSpeakWord).not.toHaveBeenCalled();
|
||||
|
||||
audio.onended?.();
|
||||
await resultPromise;
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,232 @@
|
||||
import { beforeEach, describe, expect, test, vi } from 'vitest';
|
||||
import { TTSController } from '@/services/tts/TTSController';
|
||||
import { TTSClient, TTSMessageEvent } from '@/services/tts/TTSClient';
|
||||
import { recordMeasuredDuration } from '@/services/tts/ttsDuration';
|
||||
import { FoliateView } from '@/types/view';
|
||||
|
||||
// --- Heavy clients replaced with light fakes (same pattern as the main
|
||||
// controller suite); foliate tts.js provides a REAL-shaped getSentences fake
|
||||
// that yields ranges from a jsdom document.
|
||||
|
||||
const makeMockClient = (name: string): TTSClient => ({
|
||||
name,
|
||||
initialized: true,
|
||||
init: vi.fn().mockResolvedValue(true),
|
||||
shutdown: vi.fn().mockResolvedValue(undefined),
|
||||
speak: vi.fn().mockImplementation(async function* (): AsyncIterable<TTSMessageEvent> {
|
||||
yield { code: 'end', message: 'done' };
|
||||
}),
|
||||
pause: vi.fn().mockResolvedValue(true),
|
||||
resume: vi.fn().mockResolvedValue(true),
|
||||
stop: vi.fn().mockResolvedValue(undefined),
|
||||
setPrimaryLang: vi.fn(),
|
||||
setRate: vi.fn().mockResolvedValue(undefined),
|
||||
setPitch: vi.fn().mockResolvedValue(undefined),
|
||||
setVoice: vi.fn().mockResolvedValue(undefined),
|
||||
getAllVoices: vi.fn().mockResolvedValue([]),
|
||||
getVoices: vi.fn().mockResolvedValue([]),
|
||||
getGranularities: vi.fn().mockReturnValue(['sentence']),
|
||||
supportsWordBoundaries: vi.fn().mockReturnValue(true),
|
||||
getVoiceId: vi.fn().mockReturnValue('timeline-ctrl-voice'),
|
||||
getSpeakingLang: vi.fn().mockReturnValue('en'),
|
||||
getChunkPosition: vi.fn().mockReturnValue(0.5),
|
||||
});
|
||||
|
||||
vi.mock('@/services/tts/WebSpeechClient', () => ({
|
||||
WebSpeechClient: vi.fn().mockImplementation(function (this: Record<string, unknown>) {
|
||||
Object.assign(this, makeMockClient('web-speech'));
|
||||
}),
|
||||
}));
|
||||
|
||||
vi.mock('@/services/tts/EdgeTTSClient', () => ({
|
||||
EdgeTTSClient: vi.fn().mockImplementation(function (this: Record<string, unknown>) {
|
||||
Object.assign(this, makeMockClient('edge-tts'));
|
||||
}),
|
||||
}));
|
||||
|
||||
vi.mock('@/services/tts/NativeTTSClient', () => ({
|
||||
NativeTTSClient: vi.fn().mockImplementation(function (this: Record<string, unknown>) {
|
||||
Object.assign(this, makeMockClient('native'));
|
||||
}),
|
||||
}));
|
||||
|
||||
vi.mock('@/services/tts/TTSUtils', () => ({
|
||||
TTSUtils: {
|
||||
getPreferredClient: vi.fn().mockReturnValue('edge-tts'),
|
||||
setPreferredClient: vi.fn(),
|
||||
setPreferredVoice: vi.fn(),
|
||||
getPreferredVoice: vi.fn().mockReturnValue(null),
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock('foliate-js/overlayer.js', () => ({
|
||||
Overlayer: { highlight: 'highlightFn' },
|
||||
}));
|
||||
|
||||
// Sentences of the fake section; ends avoid foliate's short-word
|
||||
// abbreviation merge so counts stay predictable.
|
||||
const S0 = 'The opening sentence of the chapter reads aloud smoothly.';
|
||||
const S1 = 'A following sentence continues the passage without pause.';
|
||||
const S2 = 'The final sentence wraps the paragraph completely together.';
|
||||
|
||||
let sectionDoc: Document;
|
||||
let sentenceRanges: Range[] = [];
|
||||
|
||||
const buildSectionDoc = () => {
|
||||
const parser = new DOMParser();
|
||||
sectionDoc = parser.parseFromString(
|
||||
`<!DOCTYPE html><html lang="en"><body><p id="p">${S0} ${S1} ${S2}</p></body></html>`,
|
||||
'text/html',
|
||||
);
|
||||
const textNode = sectionDoc.getElementById('p')!.firstChild!;
|
||||
const full = textNode.textContent!;
|
||||
sentenceRanges = [S0, S1, S2].map((s) => {
|
||||
const start = full.indexOf(s);
|
||||
const range = sectionDoc.createRange();
|
||||
range.setStart(textNode, start);
|
||||
range.setEnd(textNode, start + s.length);
|
||||
return range;
|
||||
});
|
||||
};
|
||||
|
||||
vi.mock('foliate-js/tts.js', () => ({
|
||||
TTS: vi.fn().mockImplementation(function (this: Record<string, unknown>) {
|
||||
Object.assign(this, {
|
||||
start: vi.fn().mockReturnValue('<speak>hello</speak>'),
|
||||
resume: vi.fn().mockReturnValue('<speak>hello</speak>'),
|
||||
// End-of-section after one paragraph so auto-advance terminates instead
|
||||
// of looping the controller forever in tests.
|
||||
next: vi.fn().mockReturnValue(undefined),
|
||||
prev: vi.fn().mockReturnValue(undefined),
|
||||
from: vi.fn().mockReturnValue('<speak>from</speak>'),
|
||||
setMark: vi.fn().mockImplementation(() => sentenceRanges[0]!.cloneRange()),
|
||||
getLastRange: vi.fn().mockImplementation(() => sentenceRanges[0]!.cloneRange()),
|
||||
doc: null,
|
||||
});
|
||||
}),
|
||||
getSentences: vi.fn().mockImplementation(function* () {
|
||||
for (let i = 0; i < sentenceRanges.length; i++) {
|
||||
yield { blockIndex: 0, markName: String(i), range: sentenceRanges[i]! };
|
||||
}
|
||||
}),
|
||||
}));
|
||||
|
||||
vi.mock('foliate-js/text-walker.js', () => ({ textWalker: vi.fn() }));
|
||||
|
||||
vi.mock('@/utils/ssml', () => ({
|
||||
filterSSMLWithLang: vi.fn((ssml: string) => ssml),
|
||||
parseSSMLMarks: vi.fn(() => ({
|
||||
plainText: 'hello',
|
||||
marks: [{ offset: 0, name: '0', text: 'hello', language: 'en' }],
|
||||
})),
|
||||
}));
|
||||
|
||||
vi.mock('@/utils/node', () => ({ createRejectFilter: vi.fn(() => () => 1) }));
|
||||
vi.mock('@/utils/lang', () => ({
|
||||
isValidLang: vi.fn(() => true),
|
||||
isCJKLang: vi.fn(() => false),
|
||||
}));
|
||||
|
||||
const makeView = () => {
|
||||
const contents = {
|
||||
doc: sectionDoc,
|
||||
index: 0,
|
||||
overlayer: { add: vi.fn(), remove: vi.fn() },
|
||||
};
|
||||
return {
|
||||
book: { sections: [{ createDocument: vi.fn().mockResolvedValue(sectionDoc) }] },
|
||||
renderer: { getContents: () => [contents], primaryIndex: 0 },
|
||||
language: { isCJK: false },
|
||||
getCFI: vi.fn().mockReturnValue('epubcfi(/6/2!/4/2)'),
|
||||
resolveCFI: vi.fn().mockReturnValue({ anchor: () => sentenceRanges[0] }),
|
||||
tts: null,
|
||||
} as unknown as FoliateView;
|
||||
};
|
||||
|
||||
describe('TTSController section timeline', () => {
|
||||
let controller: TTSController;
|
||||
|
||||
beforeEach(async () => {
|
||||
vi.spyOn(console, 'log').mockImplementation(() => {});
|
||||
buildSectionDoc();
|
||||
controller = new TTSController(null, makeView());
|
||||
await controller.init();
|
||||
await controller.initViewTTS(0);
|
||||
});
|
||||
|
||||
test('ensureTimeline builds lazily for the edge client and caches per section', async () => {
|
||||
const timeline = await controller.ensureTimeline();
|
||||
expect(timeline).not.toBeNull();
|
||||
expect(timeline!.length).toBe(3);
|
||||
expect(await controller.ensureTimeline()).toBe(timeline);
|
||||
});
|
||||
|
||||
test('getPlaybackInfo composes sentence position with the chunk clock', async () => {
|
||||
recordMeasuredDuration('timeline-ctrl-voice', S0, 4);
|
||||
recordMeasuredDuration('timeline-ctrl-voice', S1, 6);
|
||||
await controller.ensureTimeline();
|
||||
// getLastRange resolves to sentence 0; client chunk position is 0.5s.
|
||||
const info = controller.getPlaybackInfo();
|
||||
expect(info).not.toBeNull();
|
||||
expect(info!.position).toBeCloseTo(0.5, 5);
|
||||
expect(info!.duration).toBeGreaterThan(10);
|
||||
expect(info!.measuredFraction).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
test('getPlaybackInfo is null before the timeline is built (reserved-slot state)', () => {
|
||||
expect(controller.getPlaybackInfo()).toBeNull();
|
||||
});
|
||||
|
||||
test('getPlaybackInfo is null for non-edge clients', async () => {
|
||||
await controller.setVoice('', 'en'); // empty voice id: falls through to web client
|
||||
controller.ttsClient = controller.ttsWebClient;
|
||||
expect(await controller.ensureTimeline()).toBeNull();
|
||||
expect(controller.getPlaybackInfo()).toBeNull();
|
||||
});
|
||||
|
||||
test('seekToTime snaps to the sentence and speaks from its range while playing', async () => {
|
||||
recordMeasuredDuration('timeline-ctrl-voice', S0, 4);
|
||||
recordMeasuredDuration('timeline-ctrl-voice', S1, 6);
|
||||
await controller.ensureTimeline();
|
||||
controller.state = 'playing';
|
||||
await controller.seekToTime(5); // inside sentence 1
|
||||
const tts = controller.view.tts as unknown as { from: ReturnType<typeof vi.fn> };
|
||||
expect(tts.from).toHaveBeenCalledTimes(1);
|
||||
const arg = tts.from.mock.calls[0]![0] as Range;
|
||||
expect(arg.toString()).toBe(S1);
|
||||
});
|
||||
|
||||
test('seekToTime while paused stays paused and still navigates', async () => {
|
||||
await controller.ensureTimeline();
|
||||
controller.state = 'paused';
|
||||
await controller.seekToTime(0);
|
||||
expect(controller.state).toBe('forward-paused');
|
||||
const tts = controller.view.tts as unknown as { from: ReturnType<typeof vi.fn> };
|
||||
expect(tts.from).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('seekToTime past the end clamps to the last sentence', async () => {
|
||||
await controller.ensureTimeline();
|
||||
controller.state = 'playing';
|
||||
await controller.seekToTime(9999);
|
||||
const tts = controller.view.tts as unknown as { from: ReturnType<typeof vi.fn> };
|
||||
const arg = tts.from.mock.calls[0]![0] as Range;
|
||||
expect(arg.toString()).toBe(S2);
|
||||
});
|
||||
|
||||
test('setRate rescales the timeline', async () => {
|
||||
recordMeasuredDuration('timeline-ctrl-voice', S0, 4);
|
||||
recordMeasuredDuration('timeline-ctrl-voice', S1, 4);
|
||||
recordMeasuredDuration('timeline-ctrl-voice', S2, 4);
|
||||
const timeline = await controller.ensureTimeline();
|
||||
const before = timeline!.getDuration();
|
||||
await controller.setRate(2);
|
||||
expect(timeline!.getDuration()).toBeCloseTo(before / 2, 5);
|
||||
});
|
||||
|
||||
test('shutdown drops the timeline', async () => {
|
||||
await controller.ensureTimeline();
|
||||
await controller.shutdown();
|
||||
expect(controller.getPlaybackInfo()).toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,120 @@
|
||||
import { afterEach, describe, expect, test, vi } from 'vitest';
|
||||
|
||||
import {
|
||||
calibrateVoiceRate,
|
||||
defaultCharsPerSecond,
|
||||
estimateSentenceSeconds,
|
||||
getMeasuredDuration,
|
||||
recordMeasuredDuration,
|
||||
recordProvisionalDuration,
|
||||
} from '@/services/tts/ttsDuration';
|
||||
|
||||
const VOICE = 'en-US-AriaNeural';
|
||||
|
||||
afterEach(() => {
|
||||
localStorage.clear();
|
||||
vi.unstubAllGlobals();
|
||||
});
|
||||
|
||||
describe('measured duration store', () => {
|
||||
test('records and retrieves a duration by voice and text', () => {
|
||||
recordMeasuredDuration(VOICE, 'The quick brown fox jumps.', 2.4);
|
||||
expect(getMeasuredDuration(VOICE, 'The quick brown fox jumps.')).toBe(2.4);
|
||||
});
|
||||
|
||||
test('punctuation and case variants hit the same key', () => {
|
||||
recordMeasuredDuration(VOICE, 'Hello—world! Again…', 1.7);
|
||||
expect(getMeasuredDuration(VOICE, 'hello world again')).toBe(1.7);
|
||||
expect(getMeasuredDuration(VOICE, 'Hello, world; AGAIN')).toBe(1.7);
|
||||
});
|
||||
|
||||
test('different voices do not share durations', () => {
|
||||
recordMeasuredDuration(VOICE, 'Shared sentence text here.', 2.0);
|
||||
expect(getMeasuredDuration('fr-FR-DeniseNeural', 'Shared sentence text here.')).toBeUndefined();
|
||||
});
|
||||
|
||||
test('provisional never overwrites measured; measured overwrites provisional', () => {
|
||||
recordProvisionalDuration(VOICE, 'A provisional sentence sample.', 3.0);
|
||||
expect(getMeasuredDuration(VOICE, 'A provisional sentence sample.')).toBe(3.0);
|
||||
recordMeasuredDuration(VOICE, 'A provisional sentence sample.', 2.5);
|
||||
expect(getMeasuredDuration(VOICE, 'A provisional sentence sample.')).toBe(2.5);
|
||||
recordProvisionalDuration(VOICE, 'A provisional sentence sample.', 9.9);
|
||||
expect(getMeasuredDuration(VOICE, 'A provisional sentence sample.')).toBe(2.5);
|
||||
});
|
||||
});
|
||||
|
||||
describe('defaultCharsPerSecond', () => {
|
||||
test('CJK languages are far denser per character than Latin', () => {
|
||||
expect(defaultCharsPerSecond('zh-CN')).toBeLessThan(defaultCharsPerSecond('en-US'));
|
||||
expect(defaultCharsPerSecond('ja')).toBeLessThan(defaultCharsPerSecond('fr'));
|
||||
expect(defaultCharsPerSecond('ko')).toBeLessThan(defaultCharsPerSecond('de-DE'));
|
||||
});
|
||||
});
|
||||
|
||||
describe('estimateSentenceSeconds', () => {
|
||||
test('uses the measured duration when available', () => {
|
||||
recordMeasuredDuration(VOICE, 'A measured sentence for estimates.', 4.2);
|
||||
expect(estimateSentenceSeconds('A measured sentence for estimates.', 'en', VOICE)).toBe(4.2);
|
||||
});
|
||||
|
||||
test('falls back to the script default without calibration', () => {
|
||||
const text = 'x'.repeat(150);
|
||||
const est = estimateSentenceSeconds(text, 'en', 'uncalibrated-voice');
|
||||
expect(est).toBeCloseTo(150 / defaultCharsPerSecond('en'), 3);
|
||||
});
|
||||
|
||||
test('prefers the calibrated per-voice rate over the default', () => {
|
||||
const voice = 'calibration-voice';
|
||||
// Voice speaks 20 normalized chars per second.
|
||||
calibrateVoiceRate(voice, 'a'.repeat(40), 2);
|
||||
calibrateVoiceRate(voice, 'b'.repeat(60), 3);
|
||||
const est = estimateSentenceSeconds('c'.repeat(100), 'en', voice);
|
||||
expect(est).toBeGreaterThan(100 / 25);
|
||||
expect(est).toBeLessThan(100 / 15);
|
||||
});
|
||||
|
||||
test('calibration converges toward the observed rate', () => {
|
||||
const voice = 'converging-voice';
|
||||
for (let i = 0; i < 20; i++) {
|
||||
calibrateVoiceRate(voice, 'a'.repeat(50), 5); // 10 cps observed
|
||||
}
|
||||
const est = estimateSentenceSeconds('d'.repeat(100), 'en', voice);
|
||||
expect(est).toBeGreaterThan(100 / 12);
|
||||
expect(est).toBeLessThan(100 / 8);
|
||||
});
|
||||
|
||||
test('short texts are skipped for calibration', () => {
|
||||
const voice = 'short-skip-voice';
|
||||
calibrateVoiceRate(voice, 'hi', 60); // absurd sample must be ignored
|
||||
const est = estimateSentenceSeconds('e'.repeat(150), 'en', voice);
|
||||
expect(est).toBeCloseTo(150 / defaultCharsPerSecond('en'), 3);
|
||||
});
|
||||
|
||||
test('empty text estimates zero', () => {
|
||||
expect(estimateSentenceSeconds('', 'en', VOICE)).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('persistence', () => {
|
||||
test('per-voice calibration survives a module reload via localStorage', async () => {
|
||||
calibrateVoiceRate('persisted-voice', 'a'.repeat(50), 5);
|
||||
vi.resetModules();
|
||||
const fresh = await import('@/services/tts/ttsDuration');
|
||||
const est = fresh.estimateSentenceSeconds('f'.repeat(100), 'en', 'persisted-voice');
|
||||
expect(est).toBeGreaterThan(100 / 16);
|
||||
});
|
||||
|
||||
test('does not throw when localStorage is unavailable', () => {
|
||||
vi.stubGlobal('localStorage', {
|
||||
getItem: () => {
|
||||
throw new Error('denied');
|
||||
},
|
||||
setItem: () => {
|
||||
throw new Error('denied');
|
||||
},
|
||||
clear: () => {},
|
||||
});
|
||||
expect(() => calibrateVoiceRate('blocked-voice', 'a'.repeat(50), 5)).not.toThrow();
|
||||
expect(() => estimateSentenceSeconds('g'.repeat(100), 'en', 'blocked-voice')).not.toThrow();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,138 @@
|
||||
// Fake Web Audio primitives for jsdom tests. The fake clock is manual:
|
||||
// advanceTo(t) fires due onended callbacks in end-time order, awaiting
|
||||
// microtasks between each so schedulers and waiters can react, matching how
|
||||
// the real audio clock interleaves with the JS task queue.
|
||||
|
||||
import type {
|
||||
TTSAudioBuffer,
|
||||
TTSAudioBufferSourceNode,
|
||||
TTSAudioContext,
|
||||
} from '@/services/tts/WebAudioPlayer';
|
||||
|
||||
export class FakeAudioBuffer implements TTSAudioBuffer {
|
||||
constructor(
|
||||
public samples: Float32Array,
|
||||
public readonly sampleRate: number,
|
||||
) {}
|
||||
get length() {
|
||||
return this.samples.length;
|
||||
}
|
||||
get duration() {
|
||||
return this.samples.length / this.sampleRate;
|
||||
}
|
||||
getChannelData(_channel: number): Float32Array {
|
||||
return this.samples;
|
||||
}
|
||||
copyToChannel(source: Float32Array, _channel: number): void {
|
||||
this.samples.set(source.subarray(0, this.samples.length));
|
||||
}
|
||||
}
|
||||
|
||||
export class FakeSourceNode implements TTSAudioBufferSourceNode {
|
||||
buffer: TTSAudioBuffer | null = null;
|
||||
onended: (() => void) | null = null;
|
||||
startedAt: number | null = null;
|
||||
stopped = false;
|
||||
connected = false;
|
||||
endedFired = false;
|
||||
|
||||
constructor(private ctx: FakeAudioContext) {}
|
||||
|
||||
connect(_destination: unknown): void {
|
||||
this.connected = true;
|
||||
}
|
||||
disconnect(): void {
|
||||
this.connected = false;
|
||||
}
|
||||
start(when = 0): void {
|
||||
this.startedAt = Math.max(when, this.ctx.currentTime);
|
||||
this.ctx.sources.push(this);
|
||||
}
|
||||
stop(_when?: number): void {
|
||||
this.stopped = true;
|
||||
this.onended?.();
|
||||
}
|
||||
get endTime(): number {
|
||||
return (this.startedAt ?? 0) + (this.buffer?.duration ?? 0);
|
||||
}
|
||||
}
|
||||
|
||||
export class FakeAudioContext implements TTSAudioContext {
|
||||
// Constructed instances, newest last — client tests stub the global
|
||||
// AudioContext with this class and need a handle on the context the shared
|
||||
// singleton created internally.
|
||||
static instances: FakeAudioContext[] = [];
|
||||
|
||||
currentTime = 0;
|
||||
state = 'running';
|
||||
destination = {};
|
||||
onstatechange: (() => void) | null = null;
|
||||
sources: FakeSourceNode[] = [];
|
||||
resumeCalls = 0;
|
||||
suspendCalls = 0;
|
||||
closeCalls = 0;
|
||||
sampleRate: number;
|
||||
// Overridable resume behavior for interruption-refusal tests.
|
||||
resumeImpl: (() => void) | null = null;
|
||||
decodeImpl: (data: ArrayBuffer) => Promise<TTSAudioBuffer> = async (data) =>
|
||||
new FakeAudioBuffer(new Float32Array(data.byteLength), this.sampleRate);
|
||||
|
||||
constructor(sampleRate = 24000) {
|
||||
this.sampleRate = sampleRate;
|
||||
FakeAudioContext.instances.push(this);
|
||||
}
|
||||
|
||||
async resume(): Promise<void> {
|
||||
this.resumeCalls++;
|
||||
if (this.resumeImpl) {
|
||||
this.resumeImpl();
|
||||
} else {
|
||||
this.state = 'running';
|
||||
}
|
||||
this.onstatechange?.();
|
||||
}
|
||||
async suspend(): Promise<void> {
|
||||
this.suspendCalls++;
|
||||
this.state = 'suspended';
|
||||
this.onstatechange?.();
|
||||
}
|
||||
async close(): Promise<void> {
|
||||
this.closeCalls++;
|
||||
this.state = 'closed';
|
||||
this.onstatechange?.();
|
||||
}
|
||||
createBufferSource(): FakeSourceNode {
|
||||
return new FakeSourceNode(this);
|
||||
}
|
||||
createBuffer(_channels: number, length: number, sampleRate: number): FakeAudioBuffer {
|
||||
return new FakeAudioBuffer(new Float32Array(length), sampleRate);
|
||||
}
|
||||
decodeAudioData(data: ArrayBuffer): Promise<TTSAudioBuffer> {
|
||||
return this.decodeImpl(data);
|
||||
}
|
||||
|
||||
setState(state: string): void {
|
||||
this.state = state;
|
||||
this.onstatechange?.();
|
||||
}
|
||||
|
||||
async advanceTo(t: number): Promise<void> {
|
||||
for (;;) {
|
||||
const due = this.sources
|
||||
.filter((s) => s.startedAt !== null && !s.stopped && !s.endedFired && s.endTime <= t)
|
||||
.sort((a, b) => a.endTime - b.endTime)[0];
|
||||
if (!due) break;
|
||||
this.currentTime = Math.max(this.currentTime, due.endTime);
|
||||
due.endedFired = true;
|
||||
due.onended?.();
|
||||
await Promise.resolve();
|
||||
await Promise.resolve();
|
||||
}
|
||||
this.currentTime = Math.max(this.currentTime, t);
|
||||
await Promise.resolve();
|
||||
await Promise.resolve();
|
||||
}
|
||||
}
|
||||
|
||||
export const makeBuffer = (seconds: number, sampleRate = 24000): FakeAudioBuffer =>
|
||||
new FakeAudioBuffer(new Float32Array(Math.round(seconds * sampleRate)), sampleRate);
|
||||
@@ -0,0 +1,68 @@
|
||||
import { describe, expect, test } from 'vitest';
|
||||
|
||||
import { findSpeechBounds } from '@/services/tts/pcm';
|
||||
|
||||
const SR = 24000;
|
||||
|
||||
const makeSignal = (
|
||||
leadingSilenceSec: number,
|
||||
speechSec: number,
|
||||
trailingSilenceSec: number,
|
||||
noiseFloor = 0,
|
||||
) => {
|
||||
const total = Math.round((leadingSilenceSec + speechSec + trailingSilenceSec) * SR);
|
||||
const samples = new Float32Array(total);
|
||||
const speechStart = Math.round(leadingSilenceSec * SR);
|
||||
const speechEnd = speechStart + Math.round(speechSec * SR);
|
||||
for (let i = 0; i < total; i++) {
|
||||
if (i >= speechStart && i < speechEnd) {
|
||||
samples[i] = 0.3 * Math.sin((2 * Math.PI * 440 * i) / SR);
|
||||
} else if (noiseFloor > 0) {
|
||||
// Deterministic pseudo-noise below the detection threshold, emulating
|
||||
// MP3 decoder dither/ringing in "silent" passages.
|
||||
samples[i] = noiseFloor * Math.sin((2 * Math.PI * 1731 * i) / SR + i * 0.7);
|
||||
}
|
||||
}
|
||||
return samples;
|
||||
};
|
||||
|
||||
describe('findSpeechBounds', () => {
|
||||
test('trims leading and trailing silence with head/tail pads', () => {
|
||||
const samples = makeSignal(0.5, 1.0, 0.8);
|
||||
const { startSec, endSec } = findSpeechBounds(samples, SR);
|
||||
expect(startSec).toBeGreaterThan(0.47 - 1e-6);
|
||||
expect(startSec).toBeLessThan(0.53);
|
||||
expect(endSec).toBeGreaterThan(1.44);
|
||||
expect(endSec).toBeLessThan(1.56 + 1e-6);
|
||||
expect(endSec).toBeGreaterThan(startSec);
|
||||
});
|
||||
|
||||
test('ignores a realistic decoder noise floor in silent passages', () => {
|
||||
const samples = makeSignal(0.5, 1.0, 0.8, 0.0008);
|
||||
const { startSec, endSec } = findSpeechBounds(samples, SR);
|
||||
expect(startSec).toBeGreaterThan(0.4);
|
||||
expect(startSec).toBeLessThan(0.53);
|
||||
expect(endSec).toBeGreaterThan(1.44);
|
||||
expect(endSec).toBeLessThan(1.6);
|
||||
});
|
||||
|
||||
test('all-silence input returns the full range', () => {
|
||||
const samples = new Float32Array(SR); // 1s of zeros
|
||||
const { startSec, endSec } = findSpeechBounds(samples, SR);
|
||||
expect(startSec).toBe(0);
|
||||
expect(endSec).toBeCloseTo(1, 5);
|
||||
});
|
||||
|
||||
test('empty input returns zero bounds', () => {
|
||||
const { startSec, endSec } = findSpeechBounds(new Float32Array(0), SR);
|
||||
expect(startSec).toBe(0);
|
||||
expect(endSec).toBe(0);
|
||||
});
|
||||
|
||||
test('speech reaching the buffer edges clamps to the buffer', () => {
|
||||
const samples = makeSignal(0, 0.5, 0);
|
||||
const { startSec, endSec } = findSpeechBounds(samples, SR);
|
||||
expect(startSec).toBe(0);
|
||||
expect(endSec).toBeCloseTo(0.5, 2);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,145 @@
|
||||
import { describe, expect, test } from 'vitest';
|
||||
import { textWalker } from 'foliate-js/text-walker.js';
|
||||
import { getSentences } from 'foliate-js/tts.js';
|
||||
|
||||
import { SectionTimeline, type TimelineSentence } from '@/services/tts/SectionTimeline';
|
||||
import { recordMeasuredDuration } from '@/services/tts/ttsDuration';
|
||||
|
||||
// Note: foliate's segmenter merges sentences ending in short (<=3 letter)
|
||||
// words as suspected abbreviations, so every fixture sentence ends long.
|
||||
const SENTENCE_A0 = 'The quick brown fox jumps over the lazy hound.';
|
||||
const SENTENCE_A1 = 'A second sentence follows the first one closely.';
|
||||
const SENTENCE_B0 = 'Another paragraph starts a new block of text.';
|
||||
const SENTENCE_B1 = 'It also carries a couple of sentences inside.';
|
||||
|
||||
const makeDoc = (): Document => {
|
||||
const parser = new DOMParser();
|
||||
return parser.parseFromString(
|
||||
`<!DOCTYPE html><html lang="en"><body>` +
|
||||
`<p>${SENTENCE_A0} ${SENTENCE_A1}</p>` +
|
||||
`<p>${SENTENCE_B0} ${SENTENCE_B1}</p>` +
|
||||
`</body></html>`,
|
||||
'text/html',
|
||||
);
|
||||
};
|
||||
|
||||
const enumerate = (doc: Document): TimelineSentence[] => {
|
||||
const sentences: TimelineSentence[] = [];
|
||||
for (const { blockIndex, markName, range } of getSentences(doc, textWalker, null, 'sentence')) {
|
||||
sentences.push({ blockIndex, markName, range, text: range.toString() });
|
||||
}
|
||||
return sentences;
|
||||
};
|
||||
|
||||
describe('SectionTimeline', () => {
|
||||
test('enumerates sentences via foliate getSentences with block-scoped marks', () => {
|
||||
const sentences = enumerate(makeDoc());
|
||||
expect(sentences).toHaveLength(4);
|
||||
expect(sentences[0]!.blockIndex).toBe(0);
|
||||
expect(sentences[2]!.blockIndex).toBe(1);
|
||||
// Mark names restart per block, matching foliate's per-block mark naming.
|
||||
expect(sentences[0]!.markName).toBe(sentences[2]!.markName);
|
||||
expect(sentences[0]!.text.trim()).toBe(SENTENCE_A0);
|
||||
expect(sentences[3]!.text.trim()).toBe(SENTENCE_B1);
|
||||
});
|
||||
|
||||
test('mixes measured and estimated durations after refresh', () => {
|
||||
const voice = 'timeline-voice-mixed';
|
||||
const sentences = enumerate(makeDoc());
|
||||
const timeline = new SectionTimeline(sentences, 'en', voice);
|
||||
const estimatedTotal = timeline.getDuration();
|
||||
expect(estimatedTotal).toBeGreaterThan(0);
|
||||
|
||||
recordMeasuredDuration(voice, SENTENCE_A0, 10);
|
||||
timeline.refresh();
|
||||
const mixedTotal = timeline.getDuration();
|
||||
// Sentence A0's estimate (~3s at 15cps) was replaced by the 10s measured value.
|
||||
expect(mixedTotal).toBeGreaterThan(estimatedTotal + 5);
|
||||
expect(timeline.getMeasuredFraction()).toBeGreaterThan(0);
|
||||
expect(timeline.getMeasuredFraction()).toBeLessThan(1);
|
||||
});
|
||||
|
||||
test('positionAt sums prior durations plus the within-sentence offset', () => {
|
||||
const voice = 'timeline-voice-position';
|
||||
const sentences = enumerate(makeDoc());
|
||||
recordMeasuredDuration(voice, SENTENCE_A0, 4);
|
||||
recordMeasuredDuration(voice, SENTENCE_A1, 6);
|
||||
const timeline = new SectionTimeline(sentences, 'en', voice);
|
||||
expect(timeline.positionAt(0, 1.5)).toBeCloseTo(1.5, 5);
|
||||
expect(timeline.positionAt(1, 2)).toBeCloseTo(4 + 2, 5);
|
||||
// Clamps out-of-range indexes.
|
||||
expect(timeline.positionAt(-1, 0)).toBe(0);
|
||||
expect(timeline.positionAt(99, 0)).toBeCloseTo(timeline.getDuration(), 5);
|
||||
});
|
||||
|
||||
test('rate rescales duration and position without touching stored data', () => {
|
||||
const voice = 'timeline-voice-rate';
|
||||
const sentences = enumerate(makeDoc());
|
||||
recordMeasuredDuration(voice, SENTENCE_A0, 4);
|
||||
const timeline = new SectionTimeline(sentences, 'en', voice);
|
||||
const at1 = timeline.getDuration();
|
||||
timeline.setRate(2);
|
||||
expect(timeline.getDuration()).toBeCloseTo(at1 / 2, 5);
|
||||
expect(timeline.positionAt(1, 0)).toBeCloseTo(4 / 2, 5);
|
||||
timeline.setRate(0.5);
|
||||
expect(timeline.getDuration()).toBeCloseTo(at1 * 2, 5);
|
||||
});
|
||||
|
||||
test('sentenceAtTime maps seconds to the sentence, clamping past the end', () => {
|
||||
const voice = 'timeline-voice-seek';
|
||||
const sentences = enumerate(makeDoc());
|
||||
for (const [text, dur] of [
|
||||
[SENTENCE_A0, 4],
|
||||
[SENTENCE_A1, 6],
|
||||
[SENTENCE_B0, 5],
|
||||
[SENTENCE_B1, 5],
|
||||
] as const) {
|
||||
recordMeasuredDuration(voice, text, dur);
|
||||
}
|
||||
const timeline = new SectionTimeline(sentences, 'en', voice);
|
||||
expect(timeline.sentenceAtTime(0)?.index).toBe(0);
|
||||
expect(timeline.sentenceAtTime(4.5)?.index).toBe(1);
|
||||
expect(timeline.sentenceAtTime(11)?.index).toBe(2);
|
||||
// Past the (possibly over-estimated) end: clamp to the last sentence so a
|
||||
// lock-screen scrub past the real end is not a dead gesture.
|
||||
expect(timeline.sentenceAtTime(999)?.index).toBe(3);
|
||||
expect(timeline.sentenceAtTime(-1)?.index).toBe(0);
|
||||
// Rate applies: at 2x, 5.5 real seconds is 11 media seconds = sentence 2.
|
||||
timeline.setRate(2);
|
||||
expect(timeline.sentenceAtTime(5.5)?.index).toBe(2);
|
||||
});
|
||||
|
||||
test('sentenceAtTime returns null for an empty timeline', () => {
|
||||
const timeline = new SectionTimeline([], 'en', 'timeline-voice-empty');
|
||||
expect(timeline.sentenceAtTime(0)).toBeNull();
|
||||
expect(timeline.getDuration()).toBe(0);
|
||||
});
|
||||
|
||||
test('indexOfRange finds exact ranges, sub-ranges, and rejects foreign docs', () => {
|
||||
const doc = makeDoc();
|
||||
const sentences = enumerate(doc);
|
||||
const timeline = new SectionTimeline(sentences, 'en', 'timeline-voice-range');
|
||||
expect(timeline.indexOfRange(sentences[2]!.range)).toBe(2);
|
||||
|
||||
// A collapsed range inside sentence 3 still resolves to sentence 3.
|
||||
const sub = sentences[3]!.range.cloneRange();
|
||||
sub.collapse(false);
|
||||
expect(timeline.indexOfRange(sub)).toBe(3);
|
||||
|
||||
const foreign = makeDoc().createRange();
|
||||
expect(timeline.indexOfRange(foreign)).toBe(-1);
|
||||
});
|
||||
|
||||
test('setVoice re-estimates unmeasured entries under the new voice', () => {
|
||||
const voiceA = 'timeline-voice-a';
|
||||
const voiceB = 'timeline-voice-b';
|
||||
const sentences = enumerate(makeDoc());
|
||||
recordMeasuredDuration(voiceA, SENTENCE_A0, 20);
|
||||
const timeline = new SectionTimeline(sentences, 'en', voiceA);
|
||||
const withMeasured = timeline.getDuration();
|
||||
timeline.setVoice(voiceB);
|
||||
// Voice B has no measured data: everything estimates again.
|
||||
expect(timeline.getDuration()).toBeLessThan(withMeasured);
|
||||
expect(timeline.getMeasuredFraction()).toBe(0);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,103 @@
|
||||
import { describe, expect, test } from 'vitest';
|
||||
|
||||
import { timeStretch } from '@/services/tts/timeStretch';
|
||||
|
||||
const SR = 24000;
|
||||
|
||||
const makeSpeechLike = (seconds: number) => {
|
||||
const n = Math.round(seconds * SR);
|
||||
const out = new Float32Array(n);
|
||||
for (let i = 0; i < n; i++) {
|
||||
const t = i / SR;
|
||||
// Multi-sine with slow amplitude modulation, roughly speech-shaped.
|
||||
const env = 0.5 + 0.5 * Math.sin(2 * Math.PI * 3 * t);
|
||||
out[i] =
|
||||
env *
|
||||
(0.3 * Math.sin(2 * Math.PI * 220 * t) +
|
||||
0.2 * Math.sin(2 * Math.PI * 470 * t) +
|
||||
0.1 * Math.sin(2 * Math.PI * 910 * t));
|
||||
}
|
||||
return out;
|
||||
};
|
||||
|
||||
const makeSine = (seconds: number, freq: number) => {
|
||||
const n = Math.round(seconds * SR);
|
||||
const out = new Float32Array(n);
|
||||
for (let i = 0; i < n; i++) {
|
||||
out[i] = 0.5 * Math.sin((2 * Math.PI * freq * i) / SR);
|
||||
}
|
||||
return out;
|
||||
};
|
||||
|
||||
const zeroCrossingsPerSec = (samples: Float32Array) => {
|
||||
let crossings = 0;
|
||||
for (let i = 1; i < samples.length; i++) {
|
||||
if (samples[i - 1]! <= 0 !== samples[i]! <= 0) crossings++;
|
||||
}
|
||||
return crossings / (samples.length / SR);
|
||||
};
|
||||
|
||||
describe('timeStretch', () => {
|
||||
test('tempo 1.5 shortens output to ~1/1.5 of input length', () => {
|
||||
const input = makeSpeechLike(3);
|
||||
const output = timeStretch(input, SR, 1.5);
|
||||
const ratio = output.length / input.length;
|
||||
expect(ratio).toBeGreaterThan((1 / 1.5) * 0.95);
|
||||
expect(ratio).toBeLessThan((1 / 1.5) * 1.05);
|
||||
});
|
||||
|
||||
test('tempo 0.75 lengthens output to ~1/0.75 of input length', () => {
|
||||
const input = makeSpeechLike(3);
|
||||
const output = timeStretch(input, SR, 0.75);
|
||||
const ratio = output.length / input.length;
|
||||
expect(ratio).toBeGreaterThan((1 / 0.75) * 0.95);
|
||||
expect(ratio).toBeLessThan((1 / 0.75) * 1.05);
|
||||
});
|
||||
|
||||
test('preserves pitch: zero-crossing rate unchanged at tempo 1.5', () => {
|
||||
const input = makeSine(1, 440);
|
||||
const output = timeStretch(input, SR, 1.5);
|
||||
const inputZps = zeroCrossingsPerSec(input); // ~880
|
||||
const outputZps = zeroCrossingsPerSec(output);
|
||||
// A resampler would land near 1320; WSOLA must stay near the input rate.
|
||||
expect(outputZps).toBeGreaterThan(inputZps * 0.97);
|
||||
expect(outputZps).toBeLessThan(inputZps * 1.03);
|
||||
});
|
||||
|
||||
test('tempo 1 returns an equal-length copy, not the same reference', () => {
|
||||
const input = makeSpeechLike(1);
|
||||
const output = timeStretch(input, SR, 1);
|
||||
expect(output).not.toBe(input);
|
||||
expect(output.length).toBe(input.length);
|
||||
expect(output[1234]).toBe(input[1234]);
|
||||
});
|
||||
|
||||
test('all-zero input produces finite output (zero-energy correlation guard)', () => {
|
||||
const input = new Float32Array(SR); // 1s of digital silence
|
||||
const output = timeStretch(input, SR, 1.5);
|
||||
expect(output.length).toBeGreaterThan(0);
|
||||
expect(output.every((s) => Number.isFinite(s))).toBe(true);
|
||||
});
|
||||
|
||||
test('slow tempo 0.2 produces ~5x length with finite samples', () => {
|
||||
const input = makeSpeechLike(1);
|
||||
const output = timeStretch(input, SR, 0.2);
|
||||
const ratio = output.length / input.length;
|
||||
expect(ratio).toBeGreaterThan(5 * 0.95);
|
||||
expect(ratio).toBeLessThan(5 * 1.05);
|
||||
expect(output.every((s) => Number.isFinite(s))).toBe(true);
|
||||
});
|
||||
|
||||
test('very short input is returned as a copy (below two frames)', () => {
|
||||
const input = makeSine(0.05, 440); // 50ms < 2 x 40ms frames
|
||||
const output = timeStretch(input, SR, 2);
|
||||
expect(output.length).toBe(input.length);
|
||||
});
|
||||
|
||||
test('does not mutate its input (input may be a subarray view)', () => {
|
||||
const input = makeSpeechLike(1);
|
||||
const snapshot = input.slice();
|
||||
timeStretch(input, SR, 1.5);
|
||||
expect(input).toEqual(snapshot);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,312 @@
|
||||
import { afterEach, describe, expect, test } from 'vitest';
|
||||
|
||||
import { WebAudioPlayer, type WebAudioPlayerEvent } from '@/services/tts/WebAudioPlayer';
|
||||
import { FakeAudioContext, makeBuffer } from './tts-fake-audio';
|
||||
|
||||
const SAFETY = 0.03;
|
||||
|
||||
const setVisibility = (value: 'visible' | 'hidden') => {
|
||||
Object.defineProperty(document, 'visibilityState', {
|
||||
configurable: true,
|
||||
get: () => value,
|
||||
});
|
||||
};
|
||||
|
||||
afterEach(() => {
|
||||
setVisibility('visible');
|
||||
});
|
||||
|
||||
const setup = () => {
|
||||
const ctx = new FakeAudioContext();
|
||||
const player = new WebAudioPlayer(() => ctx);
|
||||
const events: WebAudioPlayerEvent[] = [];
|
||||
return { ctx, player, events, onEvent: (e: WebAudioPlayerEvent) => events.push(e) };
|
||||
};
|
||||
|
||||
describe('WebAudioPlayer scheduling', () => {
|
||||
test('chunks are scheduled contiguously with the requested gap', async () => {
|
||||
const { ctx, player, onEvent } = setup();
|
||||
await player.ensureContext();
|
||||
const gen = player.startSession(onEvent);
|
||||
player.scheduleChunk(gen, makeBuffer(2), { trimStartSec: 0, mediaScale: 1, gapSec: 0.5 });
|
||||
player.scheduleChunk(gen, makeBuffer(3), { trimStartSec: 0, mediaScale: 1, gapSec: 0.5 });
|
||||
expect(ctx.sources[0]!.startedAt).toBeCloseTo(SAFETY, 5);
|
||||
expect(ctx.sources[1]!.startedAt).toBeCloseTo(SAFETY + 2 + 0.5, 5);
|
||||
});
|
||||
|
||||
test('chunk-start fires at schedule for index 0 and on prior onended after', async () => {
|
||||
const { ctx, player, events, onEvent } = setup();
|
||||
await player.ensureContext();
|
||||
const gen = player.startSession(onEvent);
|
||||
player.scheduleChunk(gen, makeBuffer(2), { trimStartSec: 0, mediaScale: 1, gapSec: 0.2 });
|
||||
expect(events).toEqual([{ type: 'chunk-start', chunkIndex: 0 }]);
|
||||
player.scheduleChunk(gen, makeBuffer(1), { trimStartSec: 0, mediaScale: 1, gapSec: 0.2 });
|
||||
expect(events).toHaveLength(1);
|
||||
await ctx.advanceTo(SAFETY + 2);
|
||||
expect(events).toEqual([
|
||||
{ type: 'chunk-start', chunkIndex: 0 },
|
||||
{ type: 'chunk-start', chunkIndex: 1 },
|
||||
]);
|
||||
});
|
||||
|
||||
test('stale-generation scheduleChunk is a no-op', async () => {
|
||||
const { ctx, player, events, onEvent } = setup();
|
||||
await player.ensureContext();
|
||||
const gen = player.startSession(onEvent);
|
||||
player.startSession(() => {});
|
||||
player.scheduleChunk(gen, makeBuffer(1), { trimStartSec: 0, mediaScale: 1, gapSec: 0 });
|
||||
expect(ctx.sources).toHaveLength(0);
|
||||
expect(events).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('WebAudioPlayer backpressure', () => {
|
||||
test('third chunk waits until the first finishes while visible', async () => {
|
||||
const { ctx, player, onEvent } = setup();
|
||||
await player.ensureContext();
|
||||
const gen = player.startSession(onEvent);
|
||||
player.scheduleChunk(gen, makeBuffer(2), { trimStartSec: 0, mediaScale: 1, gapSec: 0 });
|
||||
player.scheduleChunk(gen, makeBuffer(2), { trimStartSec: 0, mediaScale: 1, gapSec: 0 });
|
||||
let resolved: boolean | null = null;
|
||||
const wait = player.waitUntilReady(gen).then((r) => {
|
||||
resolved = r;
|
||||
return r;
|
||||
});
|
||||
await Promise.resolve();
|
||||
expect(resolved).toBeNull();
|
||||
await ctx.advanceTo(SAFETY + 2);
|
||||
expect(await wait).toBe(true);
|
||||
});
|
||||
|
||||
test('hidden visibility deepens the pending-chunk budget to 5', async () => {
|
||||
setVisibility('hidden');
|
||||
const { player, onEvent } = setup();
|
||||
await player.ensureContext();
|
||||
const gen = player.startSession(onEvent);
|
||||
for (let i = 0; i < 4; i++) {
|
||||
player.scheduleChunk(gen, makeBuffer(1), { trimStartSec: 0, mediaScale: 1, gapSec: 0 });
|
||||
}
|
||||
expect(await player.waitUntilReady(gen)).toBe(true);
|
||||
});
|
||||
|
||||
test('seconds cap blocks scheduling far ahead even under the chunk budget', async () => {
|
||||
setVisibility('hidden');
|
||||
const { ctx, player, onEvent } = setup();
|
||||
await player.ensureContext();
|
||||
const gen = player.startSession(onEvent);
|
||||
player.scheduleChunk(gen, makeBuffer(30), { trimStartSec: 0, mediaScale: 1, gapSec: 0 });
|
||||
player.scheduleChunk(gen, makeBuffer(31), { trimStartSec: 0, mediaScale: 1, gapSec: 0 });
|
||||
let resolved: boolean | null = null;
|
||||
const wait = player.waitUntilReady(gen).then((r) => {
|
||||
resolved = r;
|
||||
return r;
|
||||
});
|
||||
await Promise.resolve();
|
||||
expect(resolved).toBeNull(); // 61s ahead > 60s cap, though only 2 < 5 chunks
|
||||
await ctx.advanceTo(SAFETY + 30);
|
||||
expect(await wait).toBe(true);
|
||||
});
|
||||
|
||||
test('waitUntilReady returns false for a stale generation', async () => {
|
||||
const { player, onEvent } = setup();
|
||||
await player.ensureContext();
|
||||
const gen = player.startSession(onEvent);
|
||||
player.startSession(() => {});
|
||||
expect(await player.waitUntilReady(gen)).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('WebAudioPlayer session end', () => {
|
||||
test('session-end fires after endSession + last onended', async () => {
|
||||
const { ctx, player, events, onEvent } = setup();
|
||||
await player.ensureContext();
|
||||
const gen = player.startSession(onEvent);
|
||||
player.scheduleChunk(gen, makeBuffer(1), { trimStartSec: 0, mediaScale: 1, gapSec: 0 });
|
||||
player.endSession(gen);
|
||||
expect(events.filter((e) => e.type === 'session-end')).toHaveLength(0);
|
||||
await ctx.advanceTo(SAFETY + 1);
|
||||
expect(events.filter((e) => e.type === 'session-end')).toHaveLength(1);
|
||||
});
|
||||
|
||||
test('endSession with zero scheduled chunks fires session-end synchronously', async () => {
|
||||
const { player, events, onEvent } = setup();
|
||||
await player.ensureContext();
|
||||
const gen = player.startSession(onEvent);
|
||||
player.endSession(gen);
|
||||
expect(events).toEqual([{ type: 'session-end' }]);
|
||||
});
|
||||
|
||||
test('endSession after the last onended already fired still ends the session', async () => {
|
||||
const { ctx, player, events, onEvent } = setup();
|
||||
await player.ensureContext();
|
||||
const gen = player.startSession(onEvent);
|
||||
player.scheduleChunk(gen, makeBuffer(1), { trimStartSec: 0, mediaScale: 1, gapSec: 0 });
|
||||
await ctx.advanceTo(SAFETY + 1);
|
||||
expect(events.filter((e) => e.type === 'session-end')).toHaveLength(0);
|
||||
player.endSession(gen);
|
||||
expect(events.filter((e) => e.type === 'session-end')).toHaveLength(1);
|
||||
});
|
||||
|
||||
test('session-end is emitted exactly once', async () => {
|
||||
const { ctx, player, events, onEvent } = setup();
|
||||
await player.ensureContext();
|
||||
const gen = player.startSession(onEvent);
|
||||
player.scheduleChunk(gen, makeBuffer(1), { trimStartSec: 0, mediaScale: 1, gapSec: 0 });
|
||||
player.endSession(gen);
|
||||
player.endSession(gen);
|
||||
await ctx.advanceTo(SAFETY + 1);
|
||||
await ctx.advanceTo(SAFETY + 2);
|
||||
expect(events.filter((e) => e.type === 'session-end')).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('WebAudioPlayer abort', () => {
|
||||
test('abort stops pending sources, resolves waiters false, silences events', async () => {
|
||||
const { ctx, player, events, onEvent } = setup();
|
||||
await player.ensureContext();
|
||||
const gen = player.startSession(onEvent);
|
||||
for (let i = 0; i < 3; i++) {
|
||||
player.scheduleChunk(gen, makeBuffer(2), { trimStartSec: 0, mediaScale: 1, gapSec: 0 });
|
||||
}
|
||||
const wait = player.waitUntilReady(gen);
|
||||
player.abortSession();
|
||||
expect(await wait).toBe(false);
|
||||
expect(ctx.sources.every((s) => s.stopped)).toBe(true);
|
||||
const countBefore = events.length;
|
||||
await ctx.advanceTo(100);
|
||||
expect(events).toHaveLength(countBefore);
|
||||
});
|
||||
|
||||
test('double abortSession is idempotent', async () => {
|
||||
const { player, onEvent } = setup();
|
||||
await player.ensureContext();
|
||||
const gen = player.startSession(onEvent);
|
||||
player.scheduleChunk(gen, makeBuffer(1), { trimStartSec: 0, mediaScale: 1, gapSec: 0 });
|
||||
player.abortSession();
|
||||
expect(() => player.abortSession()).not.toThrow();
|
||||
});
|
||||
|
||||
test('abort storm: 10 rapid startSession calls leave one live session, no stale events', async () => {
|
||||
const { ctx, player, onEvent } = setup();
|
||||
await player.ensureContext();
|
||||
const gens: number[] = [];
|
||||
const staleEvents: WebAudioPlayerEvent[] = [];
|
||||
for (let i = 0; i < 10; i++) {
|
||||
const gen = player.startSession(i === 9 ? onEvent : (e) => staleEvents.push(e));
|
||||
player.scheduleChunk(gen, makeBuffer(1), { trimStartSec: 0, mediaScale: 1, gapSec: 0 });
|
||||
gens.push(gen);
|
||||
}
|
||||
for (const gen of gens.slice(0, 9)) {
|
||||
player.scheduleChunk(gen, makeBuffer(1), { trimStartSec: 0, mediaScale: 1, gapSec: 0 });
|
||||
expect(await player.waitUntilReady(gen)).toBe(false);
|
||||
}
|
||||
// Stale sessions emitted only their own synchronous chunk-start 0 before
|
||||
// being aborted; nothing new after the storm.
|
||||
const staleCount = staleEvents.length;
|
||||
await ctx.advanceTo(50);
|
||||
expect(staleEvents).toHaveLength(staleCount);
|
||||
expect(await player.waitUntilReady(gens[9]!)).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('WebAudioPlayer pause/resume and interruption', () => {
|
||||
test('pauseContext suspends; resumeContext resumes', async () => {
|
||||
const { ctx, player } = setup();
|
||||
await player.ensureContext();
|
||||
player.startSession(() => {});
|
||||
await player.pauseContext();
|
||||
expect(ctx.state).toBe('suspended');
|
||||
await player.resumeContext();
|
||||
expect(ctx.state).toBe('running');
|
||||
});
|
||||
|
||||
test('auto-resumes on unexpected suspension while a session is live', async () => {
|
||||
const { ctx, player, onEvent } = setup();
|
||||
await player.ensureContext();
|
||||
const gen = player.startSession(onEvent);
|
||||
player.scheduleChunk(gen, makeBuffer(2), { trimStartSec: 0, mediaScale: 1, gapSec: 0 });
|
||||
const before = ctx.resumeCalls;
|
||||
ctx.setState('interrupted');
|
||||
await Promise.resolve();
|
||||
expect(ctx.resumeCalls).toBeGreaterThan(before);
|
||||
});
|
||||
|
||||
test('user pause suppresses auto-resume', async () => {
|
||||
const { ctx, player, onEvent } = setup();
|
||||
await player.ensureContext();
|
||||
const gen = player.startSession(onEvent);
|
||||
player.scheduleChunk(gen, makeBuffer(2), { trimStartSec: 0, mediaScale: 1, gapSec: 0 });
|
||||
await player.pauseContext();
|
||||
const before = ctx.resumeCalls;
|
||||
ctx.setState('interrupted');
|
||||
await Promise.resolve();
|
||||
expect(ctx.resumeCalls).toBe(before);
|
||||
});
|
||||
|
||||
test('resumeContext throws when the context refuses to run again', async () => {
|
||||
const { ctx, player } = setup();
|
||||
await player.ensureContext();
|
||||
await player.pauseContext();
|
||||
ctx.resumeImpl = () => {
|
||||
ctx.state = 'interrupted';
|
||||
};
|
||||
await expect(player.resumeContext()).rejects.toThrow();
|
||||
});
|
||||
});
|
||||
|
||||
describe('WebAudioPlayer playback position', () => {
|
||||
test('maps playback time through trim offset and media scale', async () => {
|
||||
const { ctx, player, onEvent } = setup();
|
||||
await player.ensureContext();
|
||||
const gen = player.startSession(onEvent);
|
||||
// 2s output chunk covering original media [0.3, 3.3) stretched 1.5x.
|
||||
player.scheduleChunk(gen, makeBuffer(2), { trimStartSec: 0.3, mediaScale: 1.5, gapSec: 0.2 });
|
||||
await ctx.advanceTo(SAFETY + 1);
|
||||
const pos = player.getPlaybackPosition(gen);
|
||||
expect(pos?.chunkIndex).toBe(0);
|
||||
expect(pos?.mediaTimeSec).toBeCloseTo(0.3 + 1 * 1.5, 3);
|
||||
});
|
||||
|
||||
test('clamps before the first chunk and in the inter-chunk gap', async () => {
|
||||
const { ctx, player, onEvent } = setup();
|
||||
await player.ensureContext();
|
||||
const gen = player.startSession(onEvent);
|
||||
player.scheduleChunk(gen, makeBuffer(1), { trimStartSec: 0.2, mediaScale: 1, gapSec: 1 });
|
||||
player.scheduleChunk(gen, makeBuffer(1), { trimStartSec: 0.1, mediaScale: 1, gapSec: 1 });
|
||||
// Before the first chunk starts.
|
||||
expect(player.getPlaybackPosition(gen)?.mediaTimeSec).toBeCloseTo(0.2, 5);
|
||||
// Inside the gap after chunk 0 (ends at SAFETY+1; next starts SAFETY+2).
|
||||
ctx.currentTime = SAFETY + 1.5;
|
||||
const pos = player.getPlaybackPosition(gen);
|
||||
expect(pos?.chunkIndex).toBe(0);
|
||||
expect(pos?.mediaTimeSec).toBeCloseTo(0.2 + 1, 5);
|
||||
});
|
||||
|
||||
test('returns null for a stale generation or empty session', async () => {
|
||||
const { player, onEvent } = setup();
|
||||
await player.ensureContext();
|
||||
const gen = player.startSession(onEvent);
|
||||
expect(player.getPlaybackPosition(gen)).toBeNull();
|
||||
player.startSession(() => {});
|
||||
expect(player.getPlaybackPosition(gen)).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('WebAudioPlayer buffers', () => {
|
||||
test('createMonoBuffer preserves samples and sample rate', async () => {
|
||||
const { player } = setup();
|
||||
const samples = new Float32Array([0.1, -0.2, 0.3, -0.4]);
|
||||
const buffer = await player.createMonoBuffer(samples, 48000);
|
||||
expect(buffer.sampleRate).toBe(48000);
|
||||
expect(buffer.length).toBe(4);
|
||||
expect(Array.from(buffer.getChannelData(0))).toEqual(Array.from(samples));
|
||||
});
|
||||
|
||||
test('decode resolves through the context decoder at its sample rate', async () => {
|
||||
const ctx = new FakeAudioContext(48000);
|
||||
const player = new WebAudioPlayer(() => ctx);
|
||||
const buffer = await player.decode(new ArrayBuffer(96000));
|
||||
expect(buffer.sampleRate).toBe(48000);
|
||||
expect(buffer.duration).toBeCloseTo(2, 5);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,33 @@
|
||||
import { describe, expect, test } from 'vitest';
|
||||
|
||||
import { formatPlaybackTime } from '@/utils/time';
|
||||
|
||||
describe('formatPlaybackTime', () => {
|
||||
test('formats minutes and seconds by default', () => {
|
||||
expect(formatPlaybackTime(0)).toBe('0:00');
|
||||
expect(formatPlaybackTime(5)).toBe('0:05');
|
||||
expect(formatPlaybackTime(62)).toBe('1:02');
|
||||
expect(formatPlaybackTime(600)).toBe('10:00');
|
||||
});
|
||||
|
||||
test('formats hours when the value reaches one hour', () => {
|
||||
expect(formatPlaybackTime(3600)).toBe('1:00:00');
|
||||
expect(formatPlaybackTime(3600 + 61)).toBe('1:01:01');
|
||||
});
|
||||
|
||||
test('forceHours pins both labels of a row to the same layout', () => {
|
||||
// Both labels use the format chosen by the TOTAL's magnitude so the row
|
||||
// never re-layouts when the elapsed time crosses an hour.
|
||||
expect(formatPlaybackTime(62, true)).toBe('0:01:02');
|
||||
});
|
||||
|
||||
test('clamps negative and non-finite input to zero', () => {
|
||||
expect(formatPlaybackTime(-5)).toBe('0:00');
|
||||
expect(formatPlaybackTime(Number.NaN)).toBe('0:00');
|
||||
expect(formatPlaybackTime(Number.POSITIVE_INFINITY)).toBe('0:00');
|
||||
});
|
||||
|
||||
test('truncates fractional seconds', () => {
|
||||
expect(formatPlaybackTime(59.9)).toBe('0:59');
|
||||
});
|
||||
});
|
||||
@@ -15,7 +15,11 @@ import TTSIcon from './TTSIcon';
|
||||
import TTSBar from './TTSBar';
|
||||
|
||||
const POPUP_WIDTH = 282;
|
||||
// The popup is fixed-height and non-scrolling: these must track the panel's
|
||||
// content or rows get clipped. The taller variant fits the scrubber row that
|
||||
// only timeline-capable (Edge) playback shows.
|
||||
const POPUP_HEIGHT = 160;
|
||||
const POPUP_HEIGHT_WITH_PROGRESS = 184;
|
||||
const POPUP_PADDING = 10;
|
||||
|
||||
interface TTSControlProps {
|
||||
@@ -46,13 +50,17 @@ const TTSControl: React.FC<TTSControlProps> = ({ bookKey, gridInsets }) => {
|
||||
const popupPadding = useResponsiveSize(POPUP_PADDING);
|
||||
const maxWidth = window.innerWidth - 2 * popupPadding;
|
||||
const popupWidth = Math.min(maxWidth, useResponsiveSize(POPUP_WIDTH));
|
||||
const popupHeight = useResponsiveSize(POPUP_HEIGHT);
|
||||
const popupBaseHeight = useResponsiveSize(POPUP_HEIGHT);
|
||||
const popupProgressHeight = useResponsiveSize(POPUP_HEIGHT_WITH_PROGRESS);
|
||||
|
||||
const tts = useTTSControl({
|
||||
bookKey,
|
||||
onRequestHidePanel: () => setShowPanel(false),
|
||||
});
|
||||
|
||||
const hasPlaybackInfo = tts.ttsClientsInited && tts.handleSupportsPlaybackInfo();
|
||||
const popupHeight = hasPlaybackInfo ? popupProgressHeight : popupBaseHeight;
|
||||
|
||||
useEffect(() => {
|
||||
if (tts.showBackToCurrentTTSLocation) {
|
||||
setShouldMountBackButton(true);
|
||||
@@ -230,6 +238,9 @@ const TTSControl: React.FC<TTSControlProps> = ({ bookKey, gridInsets }) => {
|
||||
onGetVoiceId={tts.handleGetVoiceId}
|
||||
onSelectTimeout={tts.handleSelectTimeout}
|
||||
onToogleTTSBar={tts.handleToggleTTSBar}
|
||||
onSeek={tts.handleSeekTo}
|
||||
onGetPlaybackInfo={tts.handleGetPlaybackInfo}
|
||||
hasTimeline={hasPlaybackInfo}
|
||||
/>
|
||||
</Popup>
|
||||
)}
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import clsx from 'clsx';
|
||||
import { useState, ChangeEvent, useEffect } from 'react';
|
||||
import { useState, ChangeEvent, useEffect, useRef, useCallback } from 'react';
|
||||
import { MdPlayCircle, MdPauseCircle, MdFastRewind, MdFastForward, MdAlarm } from 'react-icons/md';
|
||||
import { TbChevronCompactDown, TbChevronCompactUp } from 'react-icons/tb';
|
||||
import { RiVoiceAiFill } from 'react-icons/ri';
|
||||
@@ -11,6 +11,14 @@ import { TranslationFunc, useTranslation } from '@/hooks/useTranslation';
|
||||
import { useSettingsStore } from '@/store/settingsStore';
|
||||
import { useDefaultIconSize, useResponsiveSize } from '@/hooks/useResponsiveSize';
|
||||
import { getLanguageName } from '@/utils/lang';
|
||||
import { formatPlaybackTime } from '@/utils/time';
|
||||
import { eventDispatcher } from '@/utils/event';
|
||||
|
||||
type TTSPlaybackInfo = {
|
||||
position: number;
|
||||
duration: number;
|
||||
measuredFraction: number;
|
||||
};
|
||||
|
||||
type TTSPanelProps = {
|
||||
bookKey: string;
|
||||
@@ -27,6 +35,161 @@ type TTSPanelProps = {
|
||||
onGetVoiceId: () => string;
|
||||
onSelectTimeout: (bookKey: string, value: number) => void;
|
||||
onToogleTTSBar: () => void;
|
||||
onSeek: (seconds: number) => Promise<void>;
|
||||
onGetPlaybackInfo: () => TTSPlaybackInfo | null;
|
||||
hasTimeline: boolean;
|
||||
};
|
||||
|
||||
// Suppress poll updates briefly after a seek so the optimistic thumb never
|
||||
// snaps back while the cold-seek fetch completes.
|
||||
const SEEK_SUPPRESS_MS = 2000;
|
||||
// Small backward drifts come from estimate refinement, not playback — hold the
|
||||
// displayed position monotonic; larger jumps are deliberate (seek, chapter).
|
||||
const MONOTONIC_SLACK_SEC = 3;
|
||||
|
||||
// Chapter progress row: elapsed / thin scrubber / total. Lives between the
|
||||
// rate slider and the transport cluster; the thin range-xs track plus flanking
|
||||
// time labels keep it visually distinct from the chunky tick-labeled rate
|
||||
// slider one block up (misgrabbing THAT persists a global setting).
|
||||
const TTSProgressRow = ({
|
||||
bookKey,
|
||||
isEink,
|
||||
onSeek,
|
||||
onGetPlaybackInfo,
|
||||
}: {
|
||||
bookKey: string;
|
||||
isEink: boolean;
|
||||
onSeek: (seconds: number) => Promise<void>;
|
||||
onGetPlaybackInfo: () => TTSPlaybackInfo | null;
|
||||
}) => {
|
||||
const _ = useTranslation();
|
||||
const [info, setInfo] = useState<TTSPlaybackInfo | null>(null);
|
||||
const [stale, setStale] = useState(true);
|
||||
const [displayTotal, setDisplayTotal] = useState<number | null>(null);
|
||||
const [dragValue, setDragValue] = useState<number | null>(null);
|
||||
const dragValueRef = useRef<number | null>(null);
|
||||
const suppressUntilRef = useRef(0);
|
||||
const lastPositionRef = useRef(0);
|
||||
const keyboardCommitRef = useRef<ReturnType<typeof setTimeout> | null>(null);
|
||||
|
||||
const refresh = useCallback(() => {
|
||||
if (dragValueRef.current !== null) return;
|
||||
if (Date.now() < suppressUntilRef.current) return;
|
||||
const next = onGetPlaybackInfo();
|
||||
if (!next) {
|
||||
// Keep the last-known values rendered (disabled) across chapter
|
||||
// transitions and while the lazy timeline builds — no row blink.
|
||||
setStale(true);
|
||||
return;
|
||||
}
|
||||
let position = next.position;
|
||||
if (
|
||||
position < lastPositionRef.current &&
|
||||
lastPositionRef.current - position < MONOTONIC_SLACK_SEC
|
||||
) {
|
||||
position = lastPositionRef.current;
|
||||
}
|
||||
lastPositionRef.current = position;
|
||||
setInfo({ ...next, position });
|
||||
setStale(false);
|
||||
// Quantize the displayed total: only follow estimate drift when it moves
|
||||
// by more than 2%, so the chapter length reads stable, not twitchy.
|
||||
setDisplayTotal((prev) =>
|
||||
prev === null || Math.abs(next.duration - prev) / prev > 0.02 ? next.duration : prev,
|
||||
);
|
||||
}, [onGetPlaybackInfo]);
|
||||
|
||||
useEffect(() => {
|
||||
refresh();
|
||||
if (isEink) {
|
||||
// No 1s repaints on e-ink: follow sentence-level position events only.
|
||||
const handler = (event: CustomEvent) => {
|
||||
const detail = event.detail as { bookKey?: string; kind?: string };
|
||||
if (detail.bookKey === bookKey && detail.kind === 'sentence') refresh();
|
||||
};
|
||||
eventDispatcher.on('tts-position', handler);
|
||||
return () => eventDispatcher.off('tts-position', handler);
|
||||
}
|
||||
const interval = setInterval(refresh, 1000);
|
||||
return () => clearInterval(interval);
|
||||
}, [refresh, isEink, bookKey]);
|
||||
|
||||
const commitSeek = (seconds: number) => {
|
||||
dragValueRef.current = null;
|
||||
setDragValue(null);
|
||||
// Optimistic thumb: land on the target immediately and hold it there
|
||||
// while the seek resolves; never snap back.
|
||||
const previous = lastPositionRef.current;
|
||||
suppressUntilRef.current = Date.now() + SEEK_SUPPRESS_MS;
|
||||
lastPositionRef.current = seconds;
|
||||
setInfo((prev) => (prev ? { ...prev, position: seconds } : prev));
|
||||
onSeek(seconds).catch(() => {
|
||||
// A silent snap-back is the exact violation the optimistic thumb
|
||||
// prevents — restore visibly and say why.
|
||||
suppressUntilRef.current = 0;
|
||||
lastPositionRef.current = previous;
|
||||
setInfo((prev) => (prev ? { ...prev, position: previous } : prev));
|
||||
eventDispatcher.dispatch('toast', { message: _('Failed to seek'), type: 'error' });
|
||||
});
|
||||
};
|
||||
|
||||
const handleChange = (e: ChangeEvent<HTMLInputElement>) => {
|
||||
// React fires range onChange continuously during a drag; track the value
|
||||
// and commit only on pointer/key release.
|
||||
const value = parseFloat(e.target.value);
|
||||
dragValueRef.current = value;
|
||||
setDragValue(value);
|
||||
};
|
||||
|
||||
const handlePointerCommit = () => {
|
||||
if (dragValueRef.current !== null) commitSeek(dragValueRef.current);
|
||||
};
|
||||
|
||||
const handleKeyUp = () => {
|
||||
// Holding an arrow key must not fire a network seek per press.
|
||||
if (keyboardCommitRef.current) clearTimeout(keyboardCommitRef.current);
|
||||
keyboardCommitRef.current = setTimeout(() => {
|
||||
if (dragValueRef.current !== null) commitSeek(dragValueRef.current);
|
||||
}, 500);
|
||||
};
|
||||
|
||||
const total = displayTotal ?? info?.duration ?? 0;
|
||||
const position = Math.min(dragValue ?? info?.position ?? 0, total);
|
||||
const ready = info !== null && total > 0;
|
||||
const forceHours = total >= 3600;
|
||||
const step = Math.max(5, Math.round(total / 100)) || 5;
|
||||
const elapsedLabel = ready ? formatPlaybackTime(position, forceHours) : '--:--';
|
||||
const remainingLabel = ready
|
||||
? `-${formatPlaybackTime(Math.max(total - position, 0), forceHours)}`
|
||||
: '--:--';
|
||||
|
||||
return (
|
||||
<div className={clsx('flex w-full items-center gap-2 py-1', stale && 'opacity-60')}>
|
||||
<span className='min-w-9 text-center text-xs tabular-nums'>{elapsedLabel}</span>
|
||||
{/* Plain native range, matching the footer's Jump to Location slider:
|
||||
thin track, small thumb, visually unmistakable for the chunky rate
|
||||
slider above. */}
|
||||
<input
|
||||
className='text-base-content min-w-0 grow'
|
||||
type='range'
|
||||
min={0}
|
||||
max={total || 1}
|
||||
step={step}
|
||||
value={position}
|
||||
disabled={!ready || stale}
|
||||
onChange={handleChange}
|
||||
onPointerUp={handlePointerCommit}
|
||||
onTouchEnd={handlePointerCommit}
|
||||
onKeyUp={handleKeyUp}
|
||||
aria-label={_('Chapter progress')}
|
||||
aria-valuetext={_('{{elapsed}} of {{total}}', {
|
||||
elapsed: elapsedLabel,
|
||||
total: ready ? formatPlaybackTime(total, forceHours) : '--:--',
|
||||
})}
|
||||
/>
|
||||
<span className='min-w-10 text-center text-xs tabular-nums'>{remainingLabel}</span>
|
||||
</div>
|
||||
);
|
||||
};
|
||||
|
||||
const getTTSTimeoutOptions = (_: TranslationFunc) => {
|
||||
@@ -116,6 +279,9 @@ const TTSPanel = ({
|
||||
onGetVoiceId,
|
||||
onSelectTimeout,
|
||||
onToogleTTSBar,
|
||||
onSeek,
|
||||
onGetPlaybackInfo,
|
||||
hasTimeline,
|
||||
}: TTSPanelProps) => {
|
||||
const _ = useTranslation();
|
||||
const { envConfig } = useEnv();
|
||||
@@ -233,6 +399,14 @@ const TTSPanel = ({
|
||||
<span className='text-center'>{_('Fast')}</span>
|
||||
</div>
|
||||
</div>
|
||||
{hasTimeline && (
|
||||
<TTSProgressRow
|
||||
bookKey={bookKey}
|
||||
isEink={viewSettings?.isEink ?? false}
|
||||
onSeek={onSeek}
|
||||
onGetPlaybackInfo={onGetPlaybackInfo}
|
||||
/>
|
||||
)}
|
||||
<div className='flex items-center justify-between space-x-2'>
|
||||
<button
|
||||
onClick={() => onBackward()}
|
||||
|
||||
@@ -9,7 +9,13 @@ import { useProofreadStore } from '@/store/proofreadStore';
|
||||
import { TransformContext } from '@/services/transformers/types';
|
||||
import { proofreadTransformer } from '@/services/transformers/proofread';
|
||||
import { useTranslation } from '@/hooks/useTranslation';
|
||||
import { TTSController, TTSMark, TTSHighlightOptions, TTSVoicesGroup } from '@/services/tts';
|
||||
import {
|
||||
ensureSharedAudioContext,
|
||||
TTSController,
|
||||
TTSMark,
|
||||
TTSHighlightOptions,
|
||||
TTSVoicesGroup,
|
||||
} from '@/services/tts';
|
||||
import { TauriMediaSession } from '@/libs/mediaSession';
|
||||
import { eventDispatcher } from '@/utils/event';
|
||||
import { genSSMLRaw, parseSSMLLang } from '@/utils/ssml';
|
||||
@@ -240,6 +246,8 @@ export const useTTSControl = ({ bookKey, onRequestHidePanel }: UseTTSControlProp
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
void updateMediaSessionPosition();
|
||||
};
|
||||
|
||||
const handleHighlightMark = (e: Event) => {
|
||||
@@ -556,9 +564,7 @@ export const useTTSControl = ({ bookKey, onRequestHidePanel }: UseTTSControlProp
|
||||
previousSectionLabelRef.current = undefined;
|
||||
setTTSEnabled(bookKey, false);
|
||||
getView(bookKey)?.deselect();
|
||||
if (appService?.isMobile) {
|
||||
releaseUnblockAudio();
|
||||
}
|
||||
releaseUnblockAudio();
|
||||
|
||||
// Tear down the controller, the lock-screen media session, and the
|
||||
// background-audio session best-effort and IN PARALLEL. The controller's
|
||||
@@ -640,12 +646,17 @@ export const useTTSControl = ({ bookKey, onRequestHidePanel }: UseTTSControlProp
|
||||
}
|
||||
|
||||
try {
|
||||
// Gesture-path audio unlocks, BEFORE any network/plugin await: WebKit
|
||||
// rejects AudioContext.resume() outside the user-gesture window, and
|
||||
// speak() itself only runs after preprocessing and preload fetches.
|
||||
// The silent keep-alive element runs on ALL platforms — desktop
|
||||
// Chromium only surfaces hardware media keys while an
|
||||
// HTMLMediaElement is playing, and Edge playback no longer has one.
|
||||
unblockAudio();
|
||||
void ensureSharedAudioContext();
|
||||
if (appService?.isIOSApp) {
|
||||
await invokeUseBackgroundAudio({ enabled: true });
|
||||
}
|
||||
if (appService?.isMobile) {
|
||||
unblockAudio();
|
||||
}
|
||||
await initMediaSession();
|
||||
setTtsClientsInitialized(false);
|
||||
|
||||
@@ -704,6 +715,64 @@ export const useTTSControl = ({ bookKey, onRequestHidePanel }: UseTTSControlProp
|
||||
}
|
||||
};
|
||||
|
||||
// Push the section timeline's position/duration to the media session so the
|
||||
// lock screen shows a live scrubber. Guarded against non-finite durations
|
||||
// (empty/estimating timelines) and the position is CLAMPED, never skipped:
|
||||
// skipping would freeze the lock-screen position when estimates overshoot.
|
||||
const updateMediaSessionPosition = useCallback(async () => {
|
||||
const ttsController = ttsControllerRef.current;
|
||||
const mediaSession = mediaSessionRef.current;
|
||||
if (!ttsController || !mediaSession) return;
|
||||
await ttsController.ensureTimeline();
|
||||
const info = ttsController.getPlaybackInfo();
|
||||
if (!info || !Number.isFinite(info.duration) || info.duration <= 0) return;
|
||||
const position = Math.min(Math.max(info.position, 0), info.duration);
|
||||
if (mediaSession instanceof TauriMediaSession) {
|
||||
await mediaSession.updatePlaybackState({
|
||||
playing: ttsControllerRef.current?.state === 'playing',
|
||||
position: Math.round(position * 1000),
|
||||
duration: Math.round(info.duration * 1000),
|
||||
});
|
||||
} else if ('setPositionState' in mediaSession) {
|
||||
try {
|
||||
mediaSession.setPositionState({
|
||||
duration: info.duration,
|
||||
position,
|
||||
playbackRate: 1,
|
||||
});
|
||||
} catch {
|
||||
// Some engines reject transiently inconsistent states; the next mark
|
||||
// updates again.
|
||||
}
|
||||
}
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, []);
|
||||
|
||||
// Sentence-snapped seek used by the lock-screen scrubber and the panel.
|
||||
const handleSeekTo = useCallback(
|
||||
async (seconds: number) => {
|
||||
const ttsController = ttsControllerRef.current;
|
||||
if (!ttsController) return;
|
||||
await ttsController.seekToTime(seconds);
|
||||
void updateMediaSessionPosition();
|
||||
},
|
||||
[updateMediaSessionPosition],
|
||||
);
|
||||
|
||||
const handleGetPlaybackInfo = useCallback(() => {
|
||||
const ttsController = ttsControllerRef.current;
|
||||
if (!ttsController) return null;
|
||||
// Kick the lazy timeline build (off the playback critical path); the
|
||||
// first polls return null until it lands and the UI shows a
|
||||
// disabled/reserved row for that state.
|
||||
void ttsController.ensureTimeline();
|
||||
return ttsController.getPlaybackInfo();
|
||||
}, []);
|
||||
|
||||
const handleSupportsPlaybackInfo = useCallback(() => {
|
||||
return ttsControllerRef.current?.supportsPlaybackInfo() ?? false;
|
||||
}, []);
|
||||
|
||||
// Playback callbacks
|
||||
const handleTogglePlay = useCallback(async () => {
|
||||
const ttsController = ttsControllerRef.current;
|
||||
@@ -879,8 +948,28 @@ export const useTTSControl = ({ bookKey, onRequestHidePanel }: UseTTSControlProp
|
||||
mediaSession.setActionHandler('previoustrack', () => {
|
||||
handleBackward();
|
||||
});
|
||||
|
||||
// Seek: units differ per backend — the native plugin reports
|
||||
// milliseconds, navigator.mediaSession reports seconds. Both clamp in
|
||||
// TTSController.seekToTime via the timeline (past-the-end lands on the
|
||||
// last sentence, never a dead gesture).
|
||||
if (mediaSession instanceof TauriMediaSession) {
|
||||
mediaSession.setActionHandler('seekto', ((positionMs: number) => {
|
||||
handleSeekTo(positionMs / 1000);
|
||||
}) as (position: number) => void);
|
||||
} else {
|
||||
try {
|
||||
mediaSession.setActionHandler('seekto', (details: MediaSessionActionDetails) => {
|
||||
if (typeof details.seekTime === 'number') {
|
||||
handleSeekTo(details.seekTime);
|
||||
}
|
||||
});
|
||||
} catch {
|
||||
// 'seekto' unsupported on this engine; the in-app scrubber covers it.
|
||||
}
|
||||
}
|
||||
}
|
||||
}, [handleTogglePlay, handlePause, handleForward, handleBackward, mediaSessionRef]);
|
||||
}, [handleTogglePlay, handlePause, handleForward, handleBackward, handleSeekTo, mediaSessionRef]);
|
||||
|
||||
return {
|
||||
isPlaying,
|
||||
@@ -907,6 +996,9 @@ export const useTTSControl = ({ bookKey, onRequestHidePanel }: UseTTSControlProp
|
||||
handleSelectTimeout,
|
||||
handleToggleTTSBar,
|
||||
handleBackToCurrentTTSLocation,
|
||||
handleSeekTo,
|
||||
handleGetPlaybackInfo,
|
||||
handleSupportsPlaybackInfo,
|
||||
refreshTtsLang,
|
||||
};
|
||||
};
|
||||
|
||||
@@ -34,7 +34,13 @@ export const useTTSMediaSession = ({ bookKey }: UseTTSMediaSessionProps) => {
|
||||
unblockerAudioRef.current.preload = 'auto';
|
||||
unblockerAudioRef.current.loop = true;
|
||||
unblockerAudioRef.current.src = SILENCE_DATA;
|
||||
unblockerAudioRef.current.play();
|
||||
unblockerAudioRef.current.play().catch((err) => {
|
||||
// Autoplay policy rejects play() outside a user gesture (headless test
|
||||
// runs, programmatic TTS starts). The keep-alive is best-effort: the
|
||||
// production path calls this inside the tts-speak gesture handler, and
|
||||
// a rejection here must not surface as an unhandled rejection.
|
||||
console.warn('Keep-alive audio blocked:', err);
|
||||
});
|
||||
};
|
||||
|
||||
const releaseUnblockAudio = () => {
|
||||
|
||||
@@ -343,7 +343,7 @@ export const parseWordBoundariesHeader = (value: string | null): TTSWordBoundary
|
||||
}
|
||||
};
|
||||
|
||||
const hashPayload = (payload: EdgeTTSPayload): string => {
|
||||
export const hashTTSPayload = (payload: EdgeTTSPayload): string => {
|
||||
const base = JSON.stringify(payload);
|
||||
return md5(base);
|
||||
};
|
||||
@@ -353,12 +353,15 @@ export type EDGE_TTS_PROTOCOL = 'wss' | 'https';
|
||||
export class EdgeSpeechTTS {
|
||||
static voices = genVoiceList(EDGE_TTS_VOICES);
|
||||
private static audioCache = new LRUCache<string, Blob>(200);
|
||||
private static audioUrlCache = new LRUCache<string, string>(200, (_, url) => {
|
||||
if (url.startsWith('blob:')) {
|
||||
URL.revokeObjectURL(url);
|
||||
}
|
||||
});
|
||||
private static boundariesCache = new LRUCache<string, TTSWordBoundary[]>(200);
|
||||
// In-flight fetches keyed by payload hash. The LRU dedupes storage, not
|
||||
// requests: the playback scheduler and the preload paths race for the same
|
||||
// sentences at every paragraph start, and without this map each racer opens
|
||||
// its own WSS connection for the same audio.
|
||||
private static inflight = new Map<
|
||||
string,
|
||||
Promise<{ blob: Blob; boundaries: TTSWordBoundary[] }>
|
||||
>();
|
||||
private protocol: EDGE_TTS_PROTOCOL = 'wss';
|
||||
|
||||
constructor(protocol?: EDGE_TTS_PROTOCOL) {
|
||||
@@ -740,25 +743,42 @@ export class EdgeSpeechTTS {
|
||||
return this.#fetchEdgeSpeech(payload);
|
||||
}
|
||||
|
||||
async createAudio(
|
||||
// Fetch (or reuse) the audio blob + boundaries for a payload, deduplicating
|
||||
// both stored results (LRU) and in-flight requests (inflight map).
|
||||
async #fetchAndCache(
|
||||
payload: EdgeTTSPayload,
|
||||
): Promise<{ url: string; boundaries: TTSWordBoundary[] }> {
|
||||
const cacheKey = hashPayload(payload);
|
||||
const cachedUrl = EdgeSpeechTTS.audioUrlCache.get(cacheKey);
|
||||
if (cachedUrl) {
|
||||
return { url: cachedUrl, boundaries: EdgeSpeechTTS.boundariesCache.get(cacheKey) ?? [] };
|
||||
): Promise<{ blob: Blob; boundaries: TTSWordBoundary[] }> {
|
||||
const cacheKey = hashTTSPayload(payload);
|
||||
const cachedBlob = EdgeSpeechTTS.audioCache.get(cacheKey);
|
||||
if (cachedBlob) {
|
||||
return { blob: cachedBlob, boundaries: EdgeSpeechTTS.boundariesCache.get(cacheKey) ?? [] };
|
||||
}
|
||||
const pending = EdgeSpeechTTS.inflight.get(cacheKey);
|
||||
if (pending) return pending;
|
||||
const promise = (async () => {
|
||||
const { response, boundaries } = await this.#fetchEdgeSpeech(payload);
|
||||
const arrayBuffer = await response.arrayBuffer();
|
||||
const blob = new Blob([arrayBuffer], { type: 'audio/mpeg' });
|
||||
EdgeSpeechTTS.audioCache.set(cacheKey, blob);
|
||||
EdgeSpeechTTS.boundariesCache.set(cacheKey, boundaries);
|
||||
return { blob, boundaries };
|
||||
})();
|
||||
EdgeSpeechTTS.inflight.set(cacheKey, promise);
|
||||
try {
|
||||
return await promise;
|
||||
} finally {
|
||||
EdgeSpeechTTS.inflight.delete(cacheKey);
|
||||
}
|
||||
const { response, boundaries } = await this.#fetchEdgeSpeech(payload);
|
||||
const arrayBuffer = await response.arrayBuffer();
|
||||
const blob = new Blob([arrayBuffer], { type: 'audio/mpeg' });
|
||||
const objectUrl = URL.createObjectURL(blob);
|
||||
EdgeSpeechTTS.audioCache.set(cacheKey, blob);
|
||||
EdgeSpeechTTS.audioUrlCache.set(cacheKey, objectUrl);
|
||||
EdgeSpeechTTS.boundariesCache.set(cacheKey, boundaries);
|
||||
return { url: objectUrl, boundaries };
|
||||
}
|
||||
|
||||
async createAudioUrl(payload: EdgeTTSPayload): Promise<string> {
|
||||
return (await this.createAudio(payload)).url;
|
||||
// Audio bytes for Web Audio decoding. The cache keeps a Blob and every call
|
||||
// mints a fresh ArrayBuffer copy via blob.arrayBuffer() — WebKit's
|
||||
// decodeAudioData detaches its input, so handing out a shared buffer would
|
||||
// break replay from cache on Safari.
|
||||
async createAudioData(
|
||||
payload: EdgeTTSPayload,
|
||||
): Promise<{ data: ArrayBuffer; boundaries: TTSWordBoundary[] }> {
|
||||
const { blob, boundaries } = await this.#fetchAndCache(payload);
|
||||
return { data: await blob.arrayBuffer(), boundaries };
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2,12 +2,63 @@ import { getUserLocale } from '@/utils/misc';
|
||||
import { isSameLang } from '@/utils/lang';
|
||||
import { TTSClient, TTSMessageEvent } from './TTSClient';
|
||||
import { EdgeSpeechTTS, EdgeTTSPayload, EDGE_TTS_PROTOCOL, TTSWordBoundary } from '@/libs/edgeTTS';
|
||||
import { TTSGranularity, TTSVoice, TTSVoicesGroup } from './types';
|
||||
import { TTSGranularity, TTSMark, TTSVoice, TTSVoicesGroup } from './types';
|
||||
import { AppService } from '@/types/system';
|
||||
import { parseSSMLMarks } from '@/utils/ssml';
|
||||
import { TTSController } from './TTSController';
|
||||
import { TTSUtils } from './TTSUtils';
|
||||
import { findBoundaryIndexAtTime } from './wordHighlight';
|
||||
import { findSpeechBounds } from './pcm';
|
||||
import { timeStretch } from './timeStretch';
|
||||
import {
|
||||
calibrateVoiceRate,
|
||||
recordMeasuredDuration,
|
||||
recordProvisionalDuration,
|
||||
} from './ttsDuration';
|
||||
import { TTSAudioBuffer, WebAudioPlayer, WebAudioPlayerEvent } from './WebAudioPlayer';
|
||||
|
||||
// Playback pipeline: fetch MP3 (cached at rate 1.0) -> decode -> trim silence
|
||||
// -> WSOLA time-stretch to the playback rate -> schedule gaplessly on the
|
||||
// shared AudioContext. Marks are dispatched when a chunk becomes AUDIBLE
|
||||
// (player chunk-start events ride source onended, which keeps working with
|
||||
// the screen off), not when it is fetched — schedule-ahead would otherwise
|
||||
// run foliate's mark cursor ahead of the voice and break prev/next/resume.
|
||||
|
||||
// Natural pause between sentences, replacing Edge's baked-in ~300ms trailing
|
||||
// silence. Divided by the playback rate so pauses shrink with speed (#2033's
|
||||
// "gaps don't scale" complaint).
|
||||
const INTER_SENTENCE_GAP_SEC = 0.15;
|
||||
const TICKS_PER_SECOND = 10_000_000;
|
||||
|
||||
interface ChunkMeta {
|
||||
mark: TTSMark;
|
||||
boundaries: TTSWordBoundary[];
|
||||
trimStartSec: number;
|
||||
trimmedDurationSec: number;
|
||||
}
|
||||
|
||||
type SpeakQueueEvent =
|
||||
| { kind: 'chunk-start'; index: number }
|
||||
| { kind: 'chunk-skip'; markName: string }
|
||||
| { kind: 'session-end' }
|
||||
| { kind: 'error'; message: string };
|
||||
|
||||
class AsyncQueue<T> {
|
||||
#items: T[] = [];
|
||||
#resolvers: Array<(item: T) => void> = [];
|
||||
|
||||
push(item: T): void {
|
||||
const resolve = this.#resolvers.shift();
|
||||
if (resolve) resolve(item);
|
||||
else this.#items.push(item);
|
||||
}
|
||||
|
||||
next(): Promise<T> {
|
||||
const item = this.#items.shift();
|
||||
if (item !== undefined) return Promise.resolve(item);
|
||||
return new Promise((resolve) => this.#resolvers.push(resolve));
|
||||
}
|
||||
}
|
||||
|
||||
export class EdgeTTSClient implements TTSClient {
|
||||
name = 'edge-tts';
|
||||
@@ -23,11 +74,11 @@ export class EdgeTTSClient implements TTSClient {
|
||||
#pitch = 1.0;
|
||||
|
||||
#edgeTTS: EdgeSpeechTTS | null = null;
|
||||
#audioElement: HTMLAudioElement | null = null;
|
||||
#player = new WebAudioPlayer();
|
||||
#activeGeneration: number | null = null;
|
||||
#activeQueue: AsyncQueue<SpeakQueueEvent> | null = null;
|
||||
#chunkMeta: ChunkMeta[] = [];
|
||||
#isPlaying = false;
|
||||
#pausedAt = 0;
|
||||
#startedAt = 0;
|
||||
#fadeCompensation: number | null = null;
|
||||
#wordTrackingRafId: number | null = null;
|
||||
|
||||
constructor(controller?: TTSController, appService?: AppService | null) {
|
||||
@@ -62,24 +113,29 @@ export class EdgeTTSClient implements TTSClient {
|
||||
}
|
||||
|
||||
getPayload = (lang: string, text: string, voiceId: string) => {
|
||||
// Rate stays 1.0 so the MP3 cache is rate-independent; the playback rate
|
||||
// is applied client-side via time-stretch.
|
||||
return { lang, text, voice: voiceId, rate: 1.0, pitch: this.#pitch } as EdgeTTSPayload;
|
||||
};
|
||||
|
||||
// Edge TTS websocket requests fail intermittently; retry the preload a few times
|
||||
// before giving up so a single transient failure doesn't stall playback.
|
||||
#createAudioUrlWithRetry = async (
|
||||
// Edge TTS websocket requests fail intermittently; retry a few times before
|
||||
// giving up so a single transient failure doesn't stall playback. The
|
||||
// "No audio data received." failure is permanent for a given sentence, so it
|
||||
// rethrows immediately for the caller's skip path.
|
||||
#createAudioDataWithRetry = async (
|
||||
payload: EdgeTTSPayload,
|
||||
signal: AbortSignal,
|
||||
maxAttempts = 3,
|
||||
): Promise<string | undefined> => {
|
||||
): Promise<{ data: ArrayBuffer; boundaries: TTSWordBoundary[] } | undefined> => {
|
||||
let lastError: unknown;
|
||||
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
|
||||
if (signal.aborted) return undefined;
|
||||
try {
|
||||
return await this.#edgeTTS?.createAudioUrl(payload);
|
||||
return await this.#edgeTTS?.createAudioData(payload);
|
||||
} catch (err) {
|
||||
if (err instanceof Error && err.message === 'No audio data received.') throw err;
|
||||
lastError = err;
|
||||
console.warn(`Edge TTS preload attempt ${attempt}/${maxAttempts} failed`, err);
|
||||
console.warn(`Edge TTS fetch attempt ${attempt}/${maxAttempts} failed`, err);
|
||||
if (attempt < maxAttempts && !signal.aborted) {
|
||||
await new Promise((resolve) => setTimeout(resolve, 200 * attempt));
|
||||
}
|
||||
@@ -88,6 +144,25 @@ export class EdgeTTSClient implements TTSClient {
|
||||
throw lastError;
|
||||
};
|
||||
|
||||
#recordDurations = (
|
||||
voiceId: string,
|
||||
text: string,
|
||||
boundaries: TTSWordBoundary[],
|
||||
trimmedDurationSec?: number,
|
||||
) => {
|
||||
if (trimmedDurationSec !== undefined) {
|
||||
// Canonical: decode-time trimmed duration; also feeds the per-voice
|
||||
// speaking-rate calibration used by timeline estimates.
|
||||
recordMeasuredDuration(voiceId, text, trimmedDurationSec);
|
||||
calibrateVoiceRate(voiceId, text, trimmedDurationSec);
|
||||
return;
|
||||
}
|
||||
const last = boundaries[boundaries.length - 1];
|
||||
if (last) {
|
||||
recordProvisionalDuration(voiceId, text, (last.offset + last.duration) / TICKS_PER_SECOND);
|
||||
}
|
||||
};
|
||||
|
||||
getVoiceIdFromLang = async (lang: string) => {
|
||||
const preferredVoiceId = TTSUtils.getPreferredVoice(this.name, lang);
|
||||
const preferredVoice = this.#voices.find((v) => v.id === preferredVoiceId);
|
||||
@@ -103,179 +178,258 @@ export class EdgeTTSClient implements TTSClient {
|
||||
const { marks } = parseSSMLMarks(ssml, this.#primaryLang);
|
||||
|
||||
if (preload) {
|
||||
// preload the first 2 marks immediately and the rest in the background
|
||||
const maxImmediate = 2;
|
||||
for (let i = 0; i < Math.min(maxImmediate, marks.length); i++) {
|
||||
if (signal.aborted) break;
|
||||
const mark = marks[i]!;
|
||||
const { language: voiceLang } = mark;
|
||||
const voiceId = await this.getVoiceIdFromLang(voiceLang);
|
||||
this.#currentVoiceId = voiceId;
|
||||
try {
|
||||
await this.#createAudioUrlWithRetry(
|
||||
this.getPayload(voiceLang, mark.text, voiceId),
|
||||
signal,
|
||||
);
|
||||
} catch (err) {
|
||||
console.warn('Error preloading mark', i, err);
|
||||
}
|
||||
}
|
||||
if (marks.length > maxImmediate) {
|
||||
(async () => {
|
||||
for (let i = maxImmediate; i < marks.length; i++) {
|
||||
const mark = marks[i]!;
|
||||
try {
|
||||
if (signal.aborted) break;
|
||||
const { language: voiceLang } = mark;
|
||||
const voiceId = await this.getVoiceIdFromLang(voiceLang);
|
||||
await this.#createAudioUrlWithRetry(
|
||||
this.getPayload(voiceLang, mark.text, voiceId),
|
||||
signal,
|
||||
);
|
||||
} catch (err) {
|
||||
console.warn('Error preloading mark (bg)', i, err);
|
||||
}
|
||||
}
|
||||
})();
|
||||
}
|
||||
|
||||
yield {
|
||||
code: 'end',
|
||||
message: 'Preload finished',
|
||||
} as TTSMessageEvent;
|
||||
|
||||
yield* this.#preload(marks, signal);
|
||||
return;
|
||||
}
|
||||
|
||||
await this.stopInternal();
|
||||
// Reuse the same Audio element inside the ssml session
|
||||
if (!this.#audioElement) {
|
||||
this.#audioElement = new Audio();
|
||||
}
|
||||
const audio = this.#audioElement;
|
||||
audio.setAttribute('x-webkit-airplay', 'deny');
|
||||
audio.preload = 'auto';
|
||||
|
||||
for (const mark of marks) {
|
||||
this.controller?.dispatchSpeakMark(mark);
|
||||
let abortHandler: null | (() => void) = null;
|
||||
try {
|
||||
const { language: voiceLang } = mark;
|
||||
const voiceId = await this.getVoiceIdFromLang(voiceLang);
|
||||
this.#speakingLang = voiceLang;
|
||||
const audioResult = await this.#edgeTTS?.createAudio(
|
||||
this.getPayload(voiceLang, mark.text, voiceId),
|
||||
);
|
||||
const audioUrl = audioResult?.url;
|
||||
const boundaries = audioResult?.boundaries ?? [];
|
||||
if (signal.aborted) {
|
||||
yield { code: 'error', message: 'Aborted' } as TTSMessageEvent;
|
||||
break;
|
||||
}
|
||||
const queue = new AsyncQueue<SpeakQueueEvent>();
|
||||
const chunkMeta: ChunkMeta[] = [];
|
||||
this.#activeQueue = queue;
|
||||
this.#chunkMeta = chunkMeta;
|
||||
|
||||
yield {
|
||||
code: 'boundary',
|
||||
message: `Start chunk: ${mark.name}`,
|
||||
mark: mark.name,
|
||||
} as TTSMessageEvent;
|
||||
// startSession before ensureContext: starting a session declares playback
|
||||
// intent, clearing any lingering user-pause so the context may resume.
|
||||
const generation = this.#player.startSession((event: WebAudioPlayerEvent) => {
|
||||
if (event.type === 'chunk-start') {
|
||||
queue.push({ kind: 'chunk-start', index: event.chunkIndex });
|
||||
} else if (event.type === 'session-end') {
|
||||
queue.push({ kind: 'session-end' });
|
||||
} else {
|
||||
queue.push({ kind: 'error', message: event.message });
|
||||
}
|
||||
});
|
||||
this.#activeGeneration = generation;
|
||||
await this.#player.ensureContext();
|
||||
this.#isPlaying = true;
|
||||
|
||||
const result = await new Promise<TTSMessageEvent>((resolve) => {
|
||||
const cleanUp = () => {
|
||||
this.#stopWordTracking();
|
||||
audio.onended = null;
|
||||
audio.onerror = null;
|
||||
audio.src = '';
|
||||
};
|
||||
let resolved = false;
|
||||
const handleEnded = () => {
|
||||
if (resolved) return;
|
||||
resolved = true;
|
||||
cleanUp();
|
||||
resolve({ code: 'end', message: `Chunk finished: ${mark.name}` });
|
||||
};
|
||||
this.#runScheduler(marks, signal, generation, queue, chunkMeta);
|
||||
|
||||
abortHandler = () => {
|
||||
cleanUp();
|
||||
resolve({ code: 'error', message: 'Aborted' });
|
||||
};
|
||||
if (signal.aborted) {
|
||||
abortHandler();
|
||||
return;
|
||||
} else {
|
||||
signal.addEventListener('abort', abortHandler);
|
||||
}
|
||||
audio.onended = handleEnded;
|
||||
audio.onerror = (e) => {
|
||||
cleanUp();
|
||||
console.warn('Audio playback error:', e);
|
||||
resolve({ code: 'error', message: 'Audio playback error' });
|
||||
};
|
||||
this.#isPlaying = true;
|
||||
audio.src = audioUrl || '';
|
||||
this.#startWordTracking(audio, boundaries);
|
||||
if (!this.appService?.isLinuxApp) {
|
||||
audio.playbackRate = this.#rate;
|
||||
}
|
||||
audio
|
||||
.play()
|
||||
.then(() => {
|
||||
if (this.appService?.isLinuxApp) {
|
||||
audio.playbackRate = this.#rate;
|
||||
}
|
||||
})
|
||||
.catch((err) => {
|
||||
cleanUp();
|
||||
console.error('Failed to play audio:', err);
|
||||
resolve({ code: 'error', message: 'Playback failed: ' + err.message });
|
||||
});
|
||||
});
|
||||
yield result;
|
||||
} catch (error) {
|
||||
if (error instanceof Error && error.message === 'No audio data received.') {
|
||||
console.warn('No audio data received for:', mark.text);
|
||||
yield { code: 'end', message: `Chunk finished: ${mark.name}` } as TTSMessageEvent;
|
||||
continue;
|
||||
}
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
console.warn('TTS error for mark:', mark.text, message);
|
||||
yield { code: 'error', message } as TTSMessageEvent;
|
||||
break;
|
||||
} finally {
|
||||
if (abortHandler) {
|
||||
signal.removeEventListener('abort', abortHandler);
|
||||
let abortHandler: (() => void) | null = null;
|
||||
try {
|
||||
if (signal.aborted) {
|
||||
yield { code: 'error', message: 'Aborted' } as TTSMessageEvent;
|
||||
return;
|
||||
}
|
||||
abortHandler = () => queue.push({ kind: 'error', message: 'Aborted' });
|
||||
signal.addEventListener('abort', abortHandler);
|
||||
|
||||
for (;;) {
|
||||
const event = await queue.next();
|
||||
if (event.kind === 'chunk-start') {
|
||||
const meta = chunkMeta[event.index];
|
||||
if (!meta) continue;
|
||||
this.controller?.dispatchSpeakMark(meta.mark);
|
||||
this.#startWordTracking(generation, event.index, meta);
|
||||
yield {
|
||||
code: 'boundary',
|
||||
message: `Start chunk: ${meta.mark.name}`,
|
||||
mark: meta.mark.name,
|
||||
} as TTSMessageEvent;
|
||||
} else if (event.kind === 'chunk-skip') {
|
||||
yield {
|
||||
code: 'end',
|
||||
message: `Chunk skipped: ${event.markName}`,
|
||||
} as TTSMessageEvent;
|
||||
} else if (event.kind === 'session-end') {
|
||||
yield { code: 'end', message: 'Speak finished' } as TTSMessageEvent;
|
||||
return;
|
||||
} else {
|
||||
yield { code: 'error', message: event.message } as TTSMessageEvent;
|
||||
return;
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
// The controller aborts the signal after every successful paragraph; a
|
||||
// lingering listener would push a stale 'Aborted' into a dead queue.
|
||||
if (abortHandler) signal.removeEventListener('abort', abortHandler);
|
||||
this.#stopWordTracking();
|
||||
this.#isPlaying = false;
|
||||
if (this.#activeGeneration === generation) {
|
||||
this.#activeGeneration = null;
|
||||
this.#activeQueue = null;
|
||||
this.#player.abortSession();
|
||||
}
|
||||
}
|
||||
await this.stopInternal();
|
||||
}
|
||||
|
||||
// Follow playback time against the word-boundary metadata of the current
|
||||
// chunk and tell the controller which word is being spoken so it can
|
||||
// highlight it. Word offsets are in media time, so audio.playbackRate and
|
||||
// pause/resume (including the resume rewind on iOS) are handled naturally
|
||||
// by polling audio.currentTime.
|
||||
#startWordTracking(audio: HTMLAudioElement, boundaries: TTSWordBoundary[]) {
|
||||
async *#preload(marks: TTSMark[], signal: AbortSignal) {
|
||||
// Fetch the first couple of marks immediately and the rest in the
|
||||
// background; the in-flight dedup in EdgeSpeechTTS keeps this from racing
|
||||
// duplicate requests against the playback scheduler.
|
||||
const maxImmediate = 2;
|
||||
for (let i = 0; i < Math.min(maxImmediate, marks.length); i++) {
|
||||
if (signal.aborted) break;
|
||||
const mark = marks[i]!;
|
||||
const voiceId = await this.getVoiceIdFromLang(mark.language);
|
||||
this.#currentVoiceId = voiceId;
|
||||
try {
|
||||
const audio = await this.#createAudioDataWithRetry(
|
||||
this.getPayload(mark.language, mark.text, voiceId),
|
||||
signal,
|
||||
);
|
||||
if (audio) this.#recordDurations(voiceId, mark.text, audio.boundaries);
|
||||
} catch (err) {
|
||||
console.warn('Error preloading mark', i, err);
|
||||
}
|
||||
}
|
||||
if (marks.length > maxImmediate) {
|
||||
(async () => {
|
||||
for (let i = maxImmediate; i < marks.length; i++) {
|
||||
const mark = marks[i]!;
|
||||
try {
|
||||
if (signal.aborted) break;
|
||||
const voiceId = await this.getVoiceIdFromLang(mark.language);
|
||||
const audio = await this.#createAudioDataWithRetry(
|
||||
this.getPayload(mark.language, mark.text, voiceId),
|
||||
signal,
|
||||
);
|
||||
if (audio) this.#recordDurations(voiceId, mark.text, audio.boundaries);
|
||||
} catch (err) {
|
||||
console.warn('Error preloading mark (bg)', i, err);
|
||||
}
|
||||
}
|
||||
})();
|
||||
}
|
||||
|
||||
yield {
|
||||
code: 'end',
|
||||
message: 'Preload finished',
|
||||
} as TTSMessageEvent;
|
||||
}
|
||||
|
||||
// Detached scheduler: fetches, prepares, and schedules chunks ahead of the
|
||||
// playhead under the player's backpressure. Never throws; failures surface
|
||||
// through the event queue.
|
||||
async #runScheduler(
|
||||
marks: TTSMark[],
|
||||
signal: AbortSignal,
|
||||
generation: number,
|
||||
queue: AsyncQueue<SpeakQueueEvent>,
|
||||
chunkMeta: ChunkMeta[],
|
||||
): Promise<void> {
|
||||
const rate = this.#rate;
|
||||
try {
|
||||
for (const mark of marks) {
|
||||
if (signal.aborted || this.#activeGeneration !== generation) return;
|
||||
// Voices resolve per mark: mixed-language sections speak (and record
|
||||
// durations under) the voice actually used for each sentence.
|
||||
const voiceId = await this.getVoiceIdFromLang(mark.language);
|
||||
this.#speakingLang = mark.language;
|
||||
this.#currentVoiceId = voiceId;
|
||||
const payload = this.getPayload(mark.language, mark.text, voiceId);
|
||||
|
||||
let audio: { data: ArrayBuffer; boundaries: TTSWordBoundary[] } | undefined;
|
||||
try {
|
||||
audio = await this.#createAudioDataWithRetry(payload, signal);
|
||||
} catch (error) {
|
||||
if (error instanceof Error && error.message === 'No audio data received.') {
|
||||
console.warn('No audio data received for:', mark.text);
|
||||
queue.push({ kind: 'chunk-skip', markName: mark.name });
|
||||
continue;
|
||||
}
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
console.warn('TTS error for mark:', mark.text, message);
|
||||
queue.push({ kind: 'error', message });
|
||||
return;
|
||||
}
|
||||
if (!audio || signal.aborted || this.#activeGeneration !== generation) return;
|
||||
this.#recordDurations(voiceId, mark.text, audio.boundaries);
|
||||
|
||||
let prepared: {
|
||||
buffer: TTSAudioBuffer;
|
||||
trimStartSec: number;
|
||||
trimmedDurationSec: number;
|
||||
};
|
||||
try {
|
||||
prepared = await this.#prepareChunkBuffer(audio.data, rate);
|
||||
} catch (error) {
|
||||
// Malformed MP3 must not dead-end the session: same UX as no-audio.
|
||||
console.warn('Failed to decode TTS audio for:', mark.text, error);
|
||||
queue.push({ kind: 'chunk-skip', markName: mark.name });
|
||||
continue;
|
||||
}
|
||||
this.#recordDurations(voiceId, mark.text, audio.boundaries, prepared.trimmedDurationSec);
|
||||
|
||||
const ready = await this.#player.waitUntilReady(generation);
|
||||
if (!ready || signal.aborted) return;
|
||||
chunkMeta.push({
|
||||
mark,
|
||||
boundaries: audio.boundaries,
|
||||
trimStartSec: prepared.trimStartSec,
|
||||
trimmedDurationSec: prepared.trimmedDurationSec,
|
||||
});
|
||||
this.#player.scheduleChunk(generation, prepared.buffer, {
|
||||
trimStartSec: prepared.trimStartSec,
|
||||
mediaScale: prepared.trimmedDurationSec / prepared.buffer.duration,
|
||||
gapSec: INTER_SENTENCE_GAP_SEC / rate,
|
||||
});
|
||||
}
|
||||
if (!signal.aborted && this.#activeGeneration === generation) {
|
||||
// Fires session-end synchronously when every mark was skipped or the
|
||||
// last chunk already ended, so the session always terminates.
|
||||
this.#player.endSession(generation);
|
||||
}
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
queue.push({ kind: 'error', message });
|
||||
}
|
||||
}
|
||||
|
||||
async #prepareChunkBuffer(
|
||||
data: ArrayBuffer,
|
||||
rate: number,
|
||||
): Promise<{ buffer: TTSAudioBuffer; trimStartSec: number; trimmedDurationSec: number }> {
|
||||
// decodeAudioData resamples to the context rate (44.1/48kHz on real
|
||||
// devices, not the stream's 24kHz) — all math below must use the decoded
|
||||
// buffer's sampleRate.
|
||||
const decoded = await this.#player.decode(data);
|
||||
const sampleRate = decoded.sampleRate;
|
||||
const channel = decoded.getChannelData(0);
|
||||
const bounds = findSpeechBounds(channel, sampleRate);
|
||||
const startSample = Math.floor(bounds.startSec * sampleRate);
|
||||
const endSample = Math.min(channel.length, Math.ceil(bounds.endSec * sampleRate));
|
||||
// A subarray is a view; timeStretch never writes its input and
|
||||
// createMonoBuffer copies, so no mutation can reach the decoded buffer.
|
||||
const trimmed = channel.subarray(startSample, endSample);
|
||||
const trimmedDurationSec = trimmed.length / sampleRate;
|
||||
const samples = rate !== 1 ? timeStretch(trimmed, sampleRate, rate) : trimmed;
|
||||
const buffer = await this.#player.createMonoBuffer(samples, sampleRate);
|
||||
return { buffer, trimStartSec: startSample / sampleRate, trimmedDurationSec };
|
||||
}
|
||||
|
||||
// Poll the audio clock (visual concern only, so rAF throttling with the
|
||||
// screen off is fine) and tell the controller which word is being spoken.
|
||||
// The player reports original (rate-1.0) media time, so Edge's boundary
|
||||
// ticks need no rescaling for trim or rate.
|
||||
#startWordTracking(generation: number, chunkIndex: number, meta: ChunkMeta): void {
|
||||
this.#stopWordTracking();
|
||||
const controller = this.controller;
|
||||
if (!controller) return;
|
||||
// Always hand the words to the controller — with boundaries it highlights
|
||||
// word-by-word; with none it draws the sentence highlight that was
|
||||
// suppressed at mark dispatch (see TTSController.prepareSpeakWords).
|
||||
controller.prepareSpeakWords(boundaries.map((boundary) => boundary.text));
|
||||
if (!boundaries.length) return;
|
||||
controller.prepareSpeakWords(meta.boundaries.map((boundary) => boundary.text));
|
||||
if (!meta.boundaries.length) return;
|
||||
let lastIndex = -1;
|
||||
const tick = () => {
|
||||
const index = findBoundaryIndexAtTime(boundaries, audio.currentTime);
|
||||
if (index !== lastIndex && index >= 0) {
|
||||
lastIndex = index;
|
||||
controller.dispatchSpeakWord(index);
|
||||
const pos = this.#player.getPlaybackPosition(generation);
|
||||
// Guard the one-frame window around a transition where this tick still
|
||||
// holds the previous chunk's boundaries.
|
||||
if (pos && pos.chunkIndex === chunkIndex) {
|
||||
const index = findBoundaryIndexAtTime(meta.boundaries, pos.mediaTimeSec);
|
||||
if (index !== lastIndex && index >= 0) {
|
||||
lastIndex = index;
|
||||
controller.dispatchSpeakWord(index);
|
||||
}
|
||||
}
|
||||
this.#wordTrackingRafId = requestAnimationFrame(tick);
|
||||
};
|
||||
this.#wordTrackingRafId = requestAnimationFrame(tick);
|
||||
}
|
||||
|
||||
#stopWordTracking() {
|
||||
#stopWordTracking(): void {
|
||||
if (this.#wordTrackingRafId !== null) {
|
||||
cancelAnimationFrame(this.#wordTrackingRafId);
|
||||
this.#wordTrackingRafId = null;
|
||||
@@ -283,35 +437,16 @@ export class EdgeTTSClient implements TTSClient {
|
||||
}
|
||||
|
||||
async pause() {
|
||||
if (!this.#isPlaying || !this.#audioElement) return true;
|
||||
this.#pausedAt = this.#audioElement.currentTime - this.#startedAt;
|
||||
await this.#audioElement.pause();
|
||||
this.#isPlaying = false;
|
||||
if (!this.#isPlaying) return true;
|
||||
await this.#player.pauseContext();
|
||||
return true;
|
||||
}
|
||||
|
||||
#getFadeCompensation() {
|
||||
if (this.#fadeCompensation !== null) return this.#fadeCompensation;
|
||||
|
||||
const userAgent = navigator.userAgent;
|
||||
const isSafari = /Safari/.test(userAgent) && !/Chrome/.test(userAgent);
|
||||
const isIOS = /iPad|iPhone|iPod/.test(userAgent);
|
||||
if (isSafari || isIOS) {
|
||||
this.#fadeCompensation = 0.2;
|
||||
} else {
|
||||
this.#fadeCompensation = 0.0;
|
||||
}
|
||||
|
||||
return this.#fadeCompensation;
|
||||
}
|
||||
|
||||
async resume() {
|
||||
if (this.#isPlaying || !this.#audioElement) return true;
|
||||
const fadeCompensation = this.#getFadeCompensation();
|
||||
this.#audioElement.currentTime = Math.max(0, this.#audioElement.currentTime - fadeCompensation);
|
||||
await this.#audioElement.play();
|
||||
this.#isPlaying = true;
|
||||
this.#startedAt = this.#audioElement.currentTime - this.#pausedAt;
|
||||
// Throws when the context refuses to run again (iOS post-interruption);
|
||||
// the controller's catch stops playback visibly instead of showing
|
||||
// "playing" over silence.
|
||||
await this.#player.resumeContext();
|
||||
return true;
|
||||
}
|
||||
|
||||
@@ -322,20 +457,33 @@ export class EdgeTTSClient implements TTSClient {
|
||||
private async stopInternal() {
|
||||
this.#stopWordTracking();
|
||||
this.#isPlaying = false;
|
||||
this.#pausedAt = 0;
|
||||
this.#startedAt = 0;
|
||||
if (this.#audioElement) {
|
||||
this.#audioElement.pause();
|
||||
this.#audioElement.currentTime = 0;
|
||||
if (this.#audioElement?.onended) {
|
||||
this.#audioElement.onended(new Event('stopped'));
|
||||
}
|
||||
this.#audioElement.src = '';
|
||||
if (this.#activeGeneration !== null) {
|
||||
this.#activeGeneration = null;
|
||||
// Unblock a generator awaiting the queue; without this a stop() outside
|
||||
// the abort path would leave the consumer parked forever.
|
||||
this.#activeQueue?.push({ kind: 'error', message: 'Aborted' });
|
||||
this.#activeQueue = null;
|
||||
this.#player.abortSession();
|
||||
}
|
||||
}
|
||||
|
||||
getChunkPosition(): number | null {
|
||||
const generation = this.#activeGeneration;
|
||||
if (generation === null) return null;
|
||||
const pos = this.#player.getPlaybackPosition(generation);
|
||||
if (!pos) return null;
|
||||
const meta = this.#chunkMeta[pos.chunkIndex];
|
||||
if (!meta) return null;
|
||||
// Trim-relative and clamped: the section timeline sums TRIMMED durations,
|
||||
// while the player reports untrimmed media time (kept that way for word
|
||||
// boundaries).
|
||||
return Math.min(Math.max(pos.mediaTimeSec - meta.trimStartSec, 0), meta.trimmedDurationSec);
|
||||
}
|
||||
|
||||
async setRate(rate: number) {
|
||||
// The Edge TTS API uses rate in [0.5 .. 2.0].
|
||||
// Applied client-side via WSOLA time-stretch at schedule time; takes
|
||||
// effect on the next speak() session (the controller restarts playback on
|
||||
// rate changes).
|
||||
this.#rate = rate;
|
||||
}
|
||||
|
||||
@@ -397,8 +545,9 @@ export class EdgeTTSClient implements TTSClient {
|
||||
}
|
||||
|
||||
async shutdown(): Promise<void> {
|
||||
await this.stopInternal();
|
||||
await this.#player.shutdown();
|
||||
this.initialized = false;
|
||||
this.#audioElement = null;
|
||||
this.#voices = [];
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,150 @@
|
||||
// Virtual playback timeline over one section's sentences.
|
||||
//
|
||||
// The section is never concatenated into a real audio stream; this table plus
|
||||
// prefix-summed durations IS the "whole chapter audio" the scrubber and the
|
||||
// media session expose. Durations are stored at rate 1.0 and resolved
|
||||
// per-sentence as measured (exact, from the duration store) or estimated
|
||||
// (per-voice calibration falling back to script defaults); outputs divide by
|
||||
// the playback rate at read time so rate changes are a pure rescale.
|
||||
//
|
||||
// Timeline sentences come from foliate's getSentences enumeration, which uses
|
||||
// the same walker/filter/granularity as the live TTS instance — ranges are
|
||||
// document-ordered and 1:1 with the marks playback produces.
|
||||
|
||||
import { estimateSentenceSeconds, getMeasuredDuration } from './ttsDuration';
|
||||
|
||||
export interface TimelineSentence {
|
||||
blockIndex: number;
|
||||
markName: string;
|
||||
range: Range;
|
||||
text: string;
|
||||
}
|
||||
|
||||
export class SectionTimeline {
|
||||
#sentences: TimelineSentence[];
|
||||
#lang: string;
|
||||
#voiceId: string;
|
||||
#rate = 1;
|
||||
#durations: number[] = [];
|
||||
#measured: boolean[] = [];
|
||||
#prefix: number[] = [];
|
||||
#total = 0;
|
||||
|
||||
constructor(sentences: TimelineSentence[], lang: string, voiceId: string) {
|
||||
this.#sentences = sentences;
|
||||
this.#lang = lang;
|
||||
this.#voiceId = voiceId;
|
||||
this.refresh();
|
||||
}
|
||||
|
||||
get length(): number {
|
||||
return this.#sentences.length;
|
||||
}
|
||||
|
||||
setVoice(voiceId: string): void {
|
||||
if (voiceId === this.#voiceId) return;
|
||||
this.#voiceId = voiceId;
|
||||
this.refresh();
|
||||
}
|
||||
|
||||
setRate(rate: number): void {
|
||||
if (Number.isFinite(rate) && rate > 0) this.#rate = rate;
|
||||
}
|
||||
|
||||
// Re-pull measured durations (playback and preload record them continuously)
|
||||
// and rebuild the prefix sums.
|
||||
refresh(): void {
|
||||
const n = this.#sentences.length;
|
||||
this.#durations = new Array<number>(n);
|
||||
this.#measured = new Array<boolean>(n);
|
||||
this.#prefix = new Array<number>(n);
|
||||
let sum = 0;
|
||||
for (let i = 0; i < n; i++) {
|
||||
const sentence = this.#sentences[i]!;
|
||||
const measured = getMeasuredDuration(this.#voiceId, sentence.text);
|
||||
const duration =
|
||||
measured ?? estimateSentenceSeconds(sentence.text, this.#lang, this.#voiceId);
|
||||
this.#durations[i] = duration;
|
||||
this.#measured[i] = measured !== undefined;
|
||||
this.#prefix[i] = sum;
|
||||
sum += duration;
|
||||
}
|
||||
this.#total = sum;
|
||||
}
|
||||
|
||||
// Seconds at the current rate.
|
||||
getDuration(): number {
|
||||
return this.#total / this.#rate;
|
||||
}
|
||||
|
||||
// Duration-weighted fraction of the timeline backed by measured (exact)
|
||||
// values — the scrubber shows "~" while this is low.
|
||||
getMeasuredFraction(): number {
|
||||
if (this.#total <= 0) return 0;
|
||||
let measuredSum = 0;
|
||||
for (let i = 0; i < this.#durations.length; i++) {
|
||||
if (this.#measured[i]) measuredSum += this.#durations[i]!;
|
||||
}
|
||||
return measuredSum / this.#total;
|
||||
}
|
||||
|
||||
// Position (seconds at the current rate) of a sentence start plus an offset
|
||||
// within it, given in rate-1.0 media seconds.
|
||||
positionAt(index: number, withinMediaSec = 0): number {
|
||||
if (this.#sentences.length === 0 || index < 0) return 0;
|
||||
if (index >= this.#sentences.length) return this.getDuration();
|
||||
const clampedWithin = Math.min(Math.max(withinMediaSec, 0), this.#durations[index]!);
|
||||
return (this.#prefix[index]! + clampedWithin) / this.#rate;
|
||||
}
|
||||
|
||||
// Map seconds (at the current rate) to a sentence, clamping past-the-end
|
||||
// seeks to the last sentence so an over-estimated total is never a dead
|
||||
// gesture. Null only for an empty timeline.
|
||||
sentenceAtTime(seconds: number): { index: number; sentence: TimelineSentence } | null {
|
||||
const n = this.#sentences.length;
|
||||
if (n === 0) return null;
|
||||
const target = Math.max(0, seconds) * this.#rate;
|
||||
let low = 0;
|
||||
let high = n - 1;
|
||||
let index = n - 1;
|
||||
while (low <= high) {
|
||||
const mid = (low + high) >> 1;
|
||||
const start = this.#prefix[mid]!;
|
||||
if (start <= target) {
|
||||
index = mid;
|
||||
low = mid + 1;
|
||||
} else {
|
||||
high = mid - 1;
|
||||
}
|
||||
}
|
||||
// Binary search finds the last sentence starting at or before the target;
|
||||
// past-the-end targets land on the final sentence by construction.
|
||||
return { index, sentence: this.#sentences[index]! };
|
||||
}
|
||||
|
||||
// Locate the timeline sentence containing (or last starting at or before)
|
||||
// the given range. Returns -1 for ranges from another document (stale
|
||||
// timeline) or ranges before the first sentence.
|
||||
indexOfRange(range: Range): number {
|
||||
const n = this.#sentences.length;
|
||||
if (n === 0) return -1;
|
||||
try {
|
||||
let low = 0;
|
||||
let high = n - 1;
|
||||
let index = -1;
|
||||
while (low <= high) {
|
||||
const mid = (low + high) >> 1;
|
||||
if (this.#sentences[mid]!.range.compareBoundaryPoints(Range.START_TO_START, range) <= 0) {
|
||||
index = mid;
|
||||
low = mid + 1;
|
||||
} else {
|
||||
high = mid - 1;
|
||||
}
|
||||
}
|
||||
return index;
|
||||
} catch {
|
||||
// WrongDocumentError: the section changed under us.
|
||||
return -1;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -29,4 +29,9 @@ export interface TTSClient {
|
||||
supportsWordBoundaries(): boolean;
|
||||
getVoiceId(): string;
|
||||
getSpeakingLang(): string;
|
||||
// Playback position within the currently audible sentence, in trimmed media
|
||||
// seconds at rate 1.0, clamped to [0, sentenceDuration]. Optional: only
|
||||
// clients with a real audio clock (Edge via Web Audio) implement it; the
|
||||
// section timeline treats absence as sentence-granularity positions.
|
||||
getChunkPosition?(): number | null;
|
||||
}
|
||||
|
||||
@@ -13,6 +13,7 @@ import { createRejectFilter } from '@/utils/node';
|
||||
import { WebSpeechClient } from './WebSpeechClient';
|
||||
import { NativeTTSClient } from './NativeTTSClient';
|
||||
import { EdgeTTSClient } from './EdgeTTSClient';
|
||||
import { SectionTimeline, TimelineSentence } from './SectionTimeline';
|
||||
import { TTSUtils } from './TTSUtils';
|
||||
import { TTSClient } from './TTSClient';
|
||||
import { isValidLang } from '@/utils/lang';
|
||||
@@ -55,6 +56,31 @@ type TTSState =
|
||||
|
||||
const HIGHLIGHT_KEY = 'tts-highlight';
|
||||
|
||||
// Node filter shared by the live TTS instance and the timeline enumeration —
|
||||
// the two MUST segment identically or timeline sentences drift from marks.
|
||||
const createTTSNodeFilter = () =>
|
||||
createRejectFilter({
|
||||
tags: ['rt', 'canvas', 'br'],
|
||||
// Footnotes/endnotes are hidden in the rendered page (see the
|
||||
// `.epubtype-footnote`/`aside[epub|type]` rules in getPageLayoutStyles);
|
||||
// skip them in TTS too, including for background sections whose
|
||||
// documents are loaded without those styles.
|
||||
classes: [
|
||||
'annotationLayer',
|
||||
'epubtype-footnote',
|
||||
'duokan-footnote-content',
|
||||
'duokan-footnote-item',
|
||||
],
|
||||
attributeTokens: [
|
||||
{
|
||||
tag: 'aside',
|
||||
attribute: 'epub:type',
|
||||
tokens: ['footnote', 'endnote', 'note', 'rearnote'],
|
||||
},
|
||||
],
|
||||
contents: [{ tag: 'a', content: /^[\[\(]?[\*\d]+[\)\]]?$/ }],
|
||||
});
|
||||
|
||||
export class TTSController extends EventTarget {
|
||||
appService: AppService | null = null;
|
||||
view: FoliateView;
|
||||
@@ -71,6 +97,15 @@ export class TTSController extends EventTarget {
|
||||
|
||||
#ttsSectionIndex: number = -1;
|
||||
|
||||
// Virtual section timeline for position/duration/seek (Edge client only).
|
||||
// Built lazily OFF the playback critical path: enumerating a 2000-sentence
|
||||
// chapter must never delay first audio.
|
||||
#sectionTimeline: SectionTimeline | null = null;
|
||||
#timelineSectionIndex: number = -1;
|
||||
#currentSentenceIndex: number = -1;
|
||||
#ttsDoc: Document | null = null;
|
||||
#ttsGranularity: TTSGranularity = 'sentence';
|
||||
|
||||
// Word-level highlight state for the currently spoken chunk. Armed by a
|
||||
// successful dispatchSpeakMark, populated by prepareSpeakWords when a TTS
|
||||
// client has word-boundary metadata for the chunk.
|
||||
@@ -243,6 +278,13 @@ export class TTSController extends EventTarget {
|
||||
}
|
||||
}
|
||||
|
||||
// The section changed (or is initializing): any previous timeline maps a
|
||||
// dead document.
|
||||
this.#sectionTimeline = null;
|
||||
this.#timelineSectionIndex = -1;
|
||||
this.#currentSentenceIndex = -1;
|
||||
this.#ttsDoc = doc;
|
||||
|
||||
if (this.view.tts && this.view.tts.doc === doc) {
|
||||
return true;
|
||||
}
|
||||
@@ -254,31 +296,12 @@ export class TTSController extends EventTarget {
|
||||
if (!supportedGranularities.includes(granularity)) {
|
||||
granularity = supportedGranularities[0]!;
|
||||
}
|
||||
this.#ttsGranularity = granularity;
|
||||
|
||||
this.view.tts = new TTS(
|
||||
doc,
|
||||
textWalker,
|
||||
createRejectFilter({
|
||||
tags: ['rt', 'canvas', 'br'],
|
||||
// Footnotes/endnotes are hidden in the rendered page (see the
|
||||
// `.epubtype-footnote`/`aside[epub|type]` rules in getPageLayoutStyles);
|
||||
// skip them in TTS too, including for background sections whose
|
||||
// documents are loaded without those styles.
|
||||
classes: [
|
||||
'annotationLayer',
|
||||
'epubtype-footnote',
|
||||
'duokan-footnote-content',
|
||||
'duokan-footnote-item',
|
||||
],
|
||||
attributeTokens: [
|
||||
{
|
||||
tag: 'aside',
|
||||
attribute: 'epub:type',
|
||||
tokens: ['footnote', 'endnote', 'note', 'rearnote'],
|
||||
},
|
||||
],
|
||||
contents: [{ tag: 'a', content: /^[\[\(]?[\*\d]+[\)\]]?$/ }],
|
||||
}),
|
||||
createTTSNodeFilter(),
|
||||
this.#getHighlighter(),
|
||||
granularity,
|
||||
);
|
||||
@@ -287,6 +310,86 @@ export class TTSController extends EventTarget {
|
||||
return true;
|
||||
}
|
||||
|
||||
// Build (or return) the virtual timeline for the current section. Edge-only:
|
||||
// it is the only client with measurable audio durations and a chunk clock.
|
||||
// Callers invoke this off the playback path (panel poll, media session).
|
||||
async ensureTimeline(): Promise<SectionTimeline | null> {
|
||||
if (this.ttsClient !== this.ttsEdgeClient) return null;
|
||||
if (this.#sectionTimeline && this.#timelineSectionIndex === this.#ttsSectionIndex) {
|
||||
return this.#sectionTimeline;
|
||||
}
|
||||
const doc = this.#ttsDoc;
|
||||
if (!doc || this.#ttsSectionIndex < 0) return null;
|
||||
const { getSentences } = await import('foliate-js/tts.js');
|
||||
const { textWalker } = await import('foliate-js/text-walker.js');
|
||||
const sentences: TimelineSentence[] = [];
|
||||
for (const entry of getSentences(
|
||||
doc,
|
||||
textWalker,
|
||||
createTTSNodeFilter(),
|
||||
this.#ttsGranularity,
|
||||
)) {
|
||||
sentences.push({ ...entry, text: entry.range.toString() });
|
||||
}
|
||||
const timeline = new SectionTimeline(
|
||||
sentences,
|
||||
this.ttsLang || 'en',
|
||||
this.ttsClient.getVoiceId(),
|
||||
);
|
||||
timeline.setRate(this.ttsRate);
|
||||
this.#sectionTimeline = timeline;
|
||||
this.#timelineSectionIndex = this.#ttsSectionIndex;
|
||||
return timeline;
|
||||
}
|
||||
|
||||
// Whether the active client can ever produce a timeline (Edge only). The
|
||||
// scrubber renders a reserved disabled slot while true and info is still
|
||||
// null, and hides entirely while false.
|
||||
supportsPlaybackInfo(): boolean {
|
||||
return this.ttsClient === this.ttsEdgeClient;
|
||||
}
|
||||
|
||||
// Position/duration of the current section playback at the current rate.
|
||||
// Null while no timeline exists (non-Edge client, timeline not yet built,
|
||||
// or nothing located yet) — the UI reserves a disabled slot for that state.
|
||||
getPlaybackInfo(): { position: number; duration: number; measuredFraction: number } | null {
|
||||
if (this.ttsClient !== this.ttsEdgeClient) return null;
|
||||
const timeline = this.#sectionTimeline;
|
||||
if (!timeline || this.#timelineSectionIndex !== this.#ttsSectionIndex) return null;
|
||||
const duration = timeline.getDuration();
|
||||
if (!Number.isFinite(duration) || duration <= 0) return null;
|
||||
let index = this.#currentSentenceIndex;
|
||||
if (index < 0) {
|
||||
const range = this.view.tts?.getLastRange();
|
||||
index = range ? timeline.indexOfRange(range) : -1;
|
||||
}
|
||||
if (index < 0) return null;
|
||||
const within = this.ttsClient.getChunkPosition?.() ?? 0;
|
||||
return {
|
||||
position: timeline.positionAt(index, within),
|
||||
duration,
|
||||
measuredFraction: timeline.getMeasuredFraction(),
|
||||
};
|
||||
}
|
||||
|
||||
// Sentence-snapped seek through the same navigation machinery as prev/next:
|
||||
// foliate's from(range) returns the paragraph SSML sliced at the target
|
||||
// sentence, so highlighting, page-follow, and mark bookkeeping come free.
|
||||
async seekToTime(seconds: number): Promise<void> {
|
||||
await this.initViewTTS();
|
||||
const timeline = await this.ensureTimeline();
|
||||
if (!timeline) return;
|
||||
const target = timeline.sentenceAtTime(seconds);
|
||||
if (!target) return;
|
||||
const isPlaying = this.state === 'playing';
|
||||
await this.stop();
|
||||
if (!isPlaying) this.state = 'forward-paused';
|
||||
this.#currentSentenceIndex = target.index;
|
||||
const ssml = this.view.tts?.from(target.sentence.range);
|
||||
await this.#handleNavigationWithSSML(ssml, isPlaying);
|
||||
if (!isPlaying) this.reapplyCurrentHighlight();
|
||||
}
|
||||
|
||||
async #initTTSForNextSection(): Promise<boolean> {
|
||||
const nextIndex = this.#ttsSectionIndex + 1;
|
||||
const sections = this.view.book.sections;
|
||||
@@ -605,6 +708,7 @@ export class TTSController extends EventTarget {
|
||||
async setRate(rate: number) {
|
||||
this.state = 'setrate-paused';
|
||||
this.ttsRate = rate;
|
||||
this.#sectionTimeline?.setRate(rate);
|
||||
await this.ttsClient.setRate(this.ttsRate);
|
||||
}
|
||||
|
||||
@@ -641,6 +745,9 @@ export class TTSController extends EventTarget {
|
||||
TTSUtils.setPreferredClient(this.ttsClient.name);
|
||||
TTSUtils.setPreferredVoice(this.ttsClient.name, lang, voiceId);
|
||||
await this.ttsClient.setVoice(voiceId);
|
||||
// A different voice speaks at a different pace: re-estimate the timeline
|
||||
// under the new voice (measured durations are keyed per voice already).
|
||||
this.#sectionTimeline?.setVoice(this.ttsClient.getVoiceId());
|
||||
}
|
||||
|
||||
getVoiceId() {
|
||||
@@ -700,6 +807,12 @@ export class TTSController extends EventTarget {
|
||||
const range = this.view.tts?.setMark(mark.name);
|
||||
this.#suppressMarkHighlight = false;
|
||||
this.#speakWordsArmed = !!range;
|
||||
if (this.#sectionTimeline && range) {
|
||||
// Keep the timeline honest as measurements land, then locate the
|
||||
// audible sentence for position reporting.
|
||||
this.#sectionTimeline.refresh();
|
||||
this.#currentSentenceIndex = this.#sectionTimeline.indexOfRange(range);
|
||||
}
|
||||
const cfi = this.view.getCFI(this.#ttsSectionIndex, range);
|
||||
this.dispatchEvent(new CustomEvent('tts-highlight-mark', { detail: { cfi } }));
|
||||
this.#dispatchPosition(cfi, 'sentence');
|
||||
@@ -860,6 +973,10 @@ export class TTSController extends EventTarget {
|
||||
await this.stop();
|
||||
this.#clearHighlighter();
|
||||
this.#ttsSectionIndex = -1;
|
||||
this.#sectionTimeline = null;
|
||||
this.#timelineSectionIndex = -1;
|
||||
this.#currentSentenceIndex = -1;
|
||||
this.#ttsDoc = null;
|
||||
this.view.tts = null;
|
||||
if (this.ttsWebClient.initialized) {
|
||||
await this.ttsWebClient.shutdown();
|
||||
|
||||
@@ -0,0 +1,346 @@
|
||||
// Gapless chunk scheduler on a persistent Web Audio context.
|
||||
//
|
||||
// Sentence buffers are scheduled back-to-back into an always-running
|
||||
// AudioContext so the OS-level output stream never stops between sentences or
|
||||
// paragraphs — per-sentence track restarts are what let Bluetooth fade-in /
|
||||
// noise gates swallow the first word (#3851) and what put audible gaps
|
||||
// between sentences (#2033). Chunk transitions ride source onended callbacks
|
||||
// (background-safe when rAF and timers are throttled with the screen off);
|
||||
// word-highlight polling is the only rAF consumer, and it lives in the client.
|
||||
//
|
||||
// The real AudioContext is a MODULE-LEVEL SINGLETON shared by all player
|
||||
// instances and never closed: a fresh TTSController (and thus client+player)
|
||||
// is constructed per tts-speak, and WebKit caps live AudioContexts (~4 on
|
||||
// iOS) — per-player contexts would leak until every new one is born suspended
|
||||
// and TTS goes silent. Sessions are isolated purely by generation tokens.
|
||||
//
|
||||
// This module speaks to the context through structural interfaces so jsdom
|
||||
// tests can drive a fake clock.
|
||||
|
||||
export interface TTSAudioBuffer {
|
||||
readonly sampleRate: number;
|
||||
readonly length: number;
|
||||
readonly duration: number;
|
||||
getChannelData(channel: number): Float32Array;
|
||||
copyToChannel(source: Float32Array, channel: number): void;
|
||||
}
|
||||
|
||||
export interface TTSAudioBufferSourceNode {
|
||||
buffer: TTSAudioBuffer | null;
|
||||
onended: (() => void) | null;
|
||||
connect(destination: unknown): void;
|
||||
disconnect(): void;
|
||||
start(when?: number, offset?: number, duration?: number): void;
|
||||
stop(when?: number): void;
|
||||
}
|
||||
|
||||
export interface TTSAudioContext {
|
||||
readonly currentTime: number;
|
||||
readonly state: string; // 'running' | 'suspended' | 'interrupted' | 'closed'
|
||||
readonly destination: unknown;
|
||||
onstatechange: (() => void) | null;
|
||||
resume(): Promise<void>;
|
||||
suspend(): Promise<void>;
|
||||
close(): Promise<void>;
|
||||
createBufferSource(): TTSAudioBufferSourceNode;
|
||||
createBuffer(numberOfChannels: number, length: number, sampleRate: number): TTSAudioBuffer;
|
||||
decodeAudioData(data: ArrayBuffer): Promise<TTSAudioBuffer>;
|
||||
}
|
||||
|
||||
export interface ChunkTiming {
|
||||
// Leading trim in original (rate-1.0) media time; word boundaries live there.
|
||||
trimStartSec: number;
|
||||
// originalTrimmedDuration / outputDuration (≈ playback rate).
|
||||
mediaScale: number;
|
||||
// Silence scheduled after this chunk; the caller rate-scales it.
|
||||
gapSec: number;
|
||||
}
|
||||
|
||||
export type WebAudioPlayerEvent =
|
||||
| { type: 'chunk-start'; chunkIndex: number }
|
||||
| { type: 'session-end' }
|
||||
| { type: 'context-error'; message: string };
|
||||
|
||||
interface ScheduledChunk {
|
||||
index: number;
|
||||
source: TTSAudioBufferSourceNode;
|
||||
startTime: number;
|
||||
duration: number;
|
||||
timing: ChunkTiming;
|
||||
ended: boolean;
|
||||
}
|
||||
|
||||
interface PlayerSession {
|
||||
generation: number;
|
||||
onEvent: (event: WebAudioPlayerEvent) => void;
|
||||
chunks: ScheduledChunk[];
|
||||
nextStartTime: number;
|
||||
ended: boolean;
|
||||
endedEmitted: boolean;
|
||||
waiters: Array<(ready: boolean) => void>;
|
||||
}
|
||||
|
||||
// Small offset so start() never lands in the past between the read of
|
||||
// currentTime and the schedule call.
|
||||
const SCHEDULE_SAFETY_SEC = 0.03;
|
||||
// Screen-off JS throttling must not starve the queue between onended and the
|
||||
// next schedule, so the pending budget deepens when the page is hidden.
|
||||
const MAX_PENDING_VISIBLE = 2;
|
||||
const MAX_PENDING_HIDDEN = 5;
|
||||
// Bounds decoded PCM at slow rates (0.2x stretches a 30s sentence to 150s).
|
||||
const MAX_AHEAD_SEC = 60;
|
||||
|
||||
let sharedContext: TTSAudioContext | null = null;
|
||||
|
||||
const getSharedContext = (): TTSAudioContext => {
|
||||
if (!sharedContext) {
|
||||
sharedContext = new AudioContext() as unknown as TTSAudioContext;
|
||||
}
|
||||
return sharedContext;
|
||||
};
|
||||
|
||||
// Warm up (create + resume) the shared context. Call this synchronously in a
|
||||
// user-gesture handler: speak() itself runs after network awaits, outside
|
||||
// WebKit's gesture window, where resume() can be rejected by autoplay policy.
|
||||
export const ensureSharedAudioContext = async (): Promise<void> => {
|
||||
if (typeof AudioContext === 'undefined') return;
|
||||
try {
|
||||
const ctx = getSharedContext();
|
||||
if (ctx.state !== 'running') {
|
||||
await ctx.resume();
|
||||
}
|
||||
} catch (err) {
|
||||
console.warn('[TTS] audio context warmup failed', err);
|
||||
}
|
||||
};
|
||||
|
||||
export class WebAudioPlayer {
|
||||
#createContext: () => TTSAudioContext;
|
||||
#usesSharedContext: boolean;
|
||||
#ctx: TTSAudioContext | null = null;
|
||||
#generation = 0;
|
||||
#session: PlayerSession | null = null;
|
||||
#userPaused = false;
|
||||
|
||||
constructor(createContext?: () => TTSAudioContext) {
|
||||
this.#createContext = createContext ?? getSharedContext;
|
||||
this.#usesSharedContext = !createContext;
|
||||
}
|
||||
|
||||
async ensureContext(): Promise<TTSAudioContext> {
|
||||
if (!this.#ctx) {
|
||||
this.#ctx = this.#createContext();
|
||||
this.#ctx.onstatechange = () => this.#handleStateChange();
|
||||
}
|
||||
if (this.#ctx.state !== 'running' && !this.#userPaused) {
|
||||
await this.#ctx.resume();
|
||||
}
|
||||
return this.#ctx;
|
||||
}
|
||||
|
||||
async decode(data: ArrayBuffer): Promise<TTSAudioBuffer> {
|
||||
const ctx = await this.ensureContext();
|
||||
return ctx.decodeAudioData(data);
|
||||
}
|
||||
|
||||
async createMonoBuffer(samples: Float32Array, sampleRate: number): Promise<TTSAudioBuffer> {
|
||||
const ctx = await this.ensureContext();
|
||||
const buffer = ctx.createBuffer(1, samples.length, sampleRate);
|
||||
buffer.copyToChannel(samples, 0);
|
||||
return buffer;
|
||||
}
|
||||
|
||||
startSession(onEvent: (event: WebAudioPlayerEvent) => void): number {
|
||||
this.abortSession();
|
||||
const generation = ++this.#generation;
|
||||
this.#session = {
|
||||
generation,
|
||||
onEvent,
|
||||
chunks: [],
|
||||
nextStartTime: 0,
|
||||
ended: false,
|
||||
endedEmitted: false,
|
||||
waiters: [],
|
||||
};
|
||||
console.log(`[TTS] session ${generation} start`);
|
||||
return generation;
|
||||
}
|
||||
|
||||
scheduleChunk(generation: number, buffer: TTSAudioBuffer, timing: ChunkTiming): void {
|
||||
const session = this.#session;
|
||||
const ctx = this.#ctx;
|
||||
if (!session || session.generation !== generation || !ctx) return;
|
||||
const start = Math.max(session.nextStartTime, ctx.currentTime + SCHEDULE_SAFETY_SEC);
|
||||
const source = ctx.createBufferSource();
|
||||
source.buffer = buffer;
|
||||
source.connect(ctx.destination);
|
||||
const chunk: ScheduledChunk = {
|
||||
index: session.chunks.length,
|
||||
source,
|
||||
startTime: start,
|
||||
duration: buffer.duration,
|
||||
timing,
|
||||
ended: false,
|
||||
};
|
||||
source.onended = () => this.#handleChunkEnded(session, chunk);
|
||||
session.chunks.push(chunk);
|
||||
session.nextStartTime = start + buffer.duration + Math.max(0, timing.gapSec);
|
||||
source.start(start);
|
||||
console.log(
|
||||
`[TTS] schedule ${generation}:${chunk.index} at ${start.toFixed(2)} dur ${buffer.duration.toFixed(2)}`,
|
||||
);
|
||||
if (chunk.index === 0) {
|
||||
session.onEvent({ type: 'chunk-start', chunkIndex: 0 });
|
||||
}
|
||||
}
|
||||
|
||||
endSession(generation: number): void {
|
||||
const session = this.#session;
|
||||
if (!session || session.generation !== generation) return;
|
||||
session.ended = true;
|
||||
// Fires synchronously when nothing is unfinished: a session whose marks
|
||||
// were all skipped (zero chunks) or whose last onended beat endSession
|
||||
// must still end, or auto-advance dead-ends with controls stuck playing.
|
||||
this.#maybeEmitSessionEnd(session);
|
||||
}
|
||||
|
||||
abortSession(): void {
|
||||
const session = this.#session;
|
||||
if (!session) return;
|
||||
this.#session = null;
|
||||
for (const chunk of session.chunks) {
|
||||
chunk.source.onended = null;
|
||||
try {
|
||||
chunk.source.stop();
|
||||
} catch {
|
||||
// Sources that never started or already ended throw; irrelevant here.
|
||||
}
|
||||
try {
|
||||
chunk.source.disconnect();
|
||||
} catch {
|
||||
// Ignore repeated disconnects.
|
||||
}
|
||||
}
|
||||
const waiters = session.waiters;
|
||||
session.waiters = [];
|
||||
for (const waiter of waiters) waiter(false);
|
||||
console.log(`[TTS] session ${session.generation} abort`);
|
||||
}
|
||||
|
||||
async waitUntilReady(generation: number): Promise<boolean> {
|
||||
for (;;) {
|
||||
const session = this.#session;
|
||||
if (!session || session.generation !== generation) return false;
|
||||
if (this.#isReadyForMore(session)) return true;
|
||||
const ready = await new Promise<boolean>((resolve) => {
|
||||
session.waiters.push(resolve);
|
||||
});
|
||||
if (!ready) return false;
|
||||
}
|
||||
}
|
||||
|
||||
async pauseContext(): Promise<void> {
|
||||
this.#userPaused = true;
|
||||
if (this.#ctx && this.#ctx.state === 'running') {
|
||||
await this.#ctx.suspend();
|
||||
}
|
||||
}
|
||||
|
||||
async resumeContext(): Promise<void> {
|
||||
this.#userPaused = false;
|
||||
const ctx = this.#ctx;
|
||||
if (!ctx) return;
|
||||
await ctx.resume();
|
||||
if (ctx.state !== 'running') {
|
||||
// iOS can refuse to leave 'interrupted' (e.g. right after a phone
|
||||
// call); fail loudly so the controller stops visibly instead of
|
||||
// showing "playing" over silence.
|
||||
throw new Error(`AudioContext failed to resume (state: ${ctx.state})`);
|
||||
}
|
||||
}
|
||||
|
||||
isUserPaused(): boolean {
|
||||
return this.#userPaused;
|
||||
}
|
||||
|
||||
getPlaybackPosition(generation: number): { chunkIndex: number; mediaTimeSec: number } | null {
|
||||
const session = this.#session;
|
||||
const ctx = this.#ctx;
|
||||
if (!session || session.generation !== generation || !ctx) return null;
|
||||
const first = session.chunks[0];
|
||||
if (!first) return null;
|
||||
const t = ctx.currentTime;
|
||||
let active = first;
|
||||
for (const chunk of session.chunks) {
|
||||
if (chunk.startTime <= t) active = chunk;
|
||||
else break;
|
||||
}
|
||||
const within = Math.min(Math.max(t - active.startTime, 0), active.duration);
|
||||
return {
|
||||
chunkIndex: active.index,
|
||||
mediaTimeSec: active.timing.trimStartSec + within * active.timing.mediaScale,
|
||||
};
|
||||
}
|
||||
|
||||
async shutdown(): Promise<void> {
|
||||
this.abortSession();
|
||||
if (this.#ctx && !this.#usesSharedContext) {
|
||||
// Test-injected contexts are owned by this player; the shared context
|
||||
// stays alive for the whole page (see module comment).
|
||||
await this.#ctx.close().catch(() => {});
|
||||
}
|
||||
this.#ctx = null;
|
||||
}
|
||||
|
||||
#isReadyForMore(session: PlayerSession): boolean {
|
||||
const unfinished = session.chunks.reduce((n, c) => n + (c.ended ? 0 : 1), 0);
|
||||
const limit =
|
||||
typeof document !== 'undefined' && document.visibilityState === 'hidden'
|
||||
? MAX_PENDING_HIDDEN
|
||||
: MAX_PENDING_VISIBLE;
|
||||
if (unfinished >= limit) return false;
|
||||
if (this.#ctx && session.chunks.length > 0) {
|
||||
const aheadSec = session.nextStartTime - this.#ctx.currentTime;
|
||||
if (aheadSec >= MAX_AHEAD_SEC) return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
#handleChunkEnded(session: PlayerSession, chunk: ScheduledChunk): void {
|
||||
if (this.#session !== session) return;
|
||||
chunk.ended = true;
|
||||
const waiters = session.waiters;
|
||||
session.waiters = [];
|
||||
for (const waiter of waiters) waiter(true);
|
||||
const next = session.chunks[chunk.index + 1];
|
||||
if (next) {
|
||||
session.onEvent({ type: 'chunk-start', chunkIndex: next.index });
|
||||
}
|
||||
this.#maybeEmitSessionEnd(session);
|
||||
}
|
||||
|
||||
#maybeEmitSessionEnd(session: PlayerSession): void {
|
||||
if (!session.ended || session.endedEmitted) return;
|
||||
if (session.chunks.some((c) => !c.ended)) return;
|
||||
session.endedEmitted = true;
|
||||
session.onEvent({ type: 'session-end' });
|
||||
}
|
||||
|
||||
#handleStateChange(): void {
|
||||
const ctx = this.#ctx;
|
||||
if (!ctx) return;
|
||||
if (ctx.state === 'running' || this.#userPaused) return;
|
||||
if (!this.#session) return;
|
||||
// Unexpected suspension (iOS 'interrupted', route change) during live
|
||||
// playback: try to keep going. If the OS refuses, the next user action
|
||||
// surfaces the failure through resumeContext().
|
||||
console.log(`[TTS] audio context ${ctx.state}; attempting auto-resume`);
|
||||
ctx.resume().catch((err) => {
|
||||
console.warn('[TTS] audio context auto-resume failed', err);
|
||||
this.#session?.onEvent({
|
||||
type: 'context-error',
|
||||
message: `AudioContext ${ctx.state}`,
|
||||
});
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -5,3 +5,5 @@ export * from './EdgeTTSClient';
|
||||
export * from './NativeTTSClient';
|
||||
export * from './TTSController';
|
||||
export * from './TTSData';
|
||||
export { ensureSharedAudioContext } from './WebAudioPlayer';
|
||||
export { SectionTimeline } from './SectionTimeline';
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
// Pure PCM helpers for the Web Audio TTS pipeline.
|
||||
//
|
||||
// Decoded MP3 "silence" is dithered ringing (roughly 1e-4 to 1e-3 amplitude),
|
||||
// not zeros, so speech detection uses an amplitude threshold rather than an
|
||||
// exact-zero test.
|
||||
|
||||
export interface SpeechBounds {
|
||||
startSec: number;
|
||||
endSec: number;
|
||||
}
|
||||
|
||||
// ~-46 dBFS: above decoder dither/ringing, below any audible speech onset.
|
||||
const DEFAULT_SILENCE_THRESHOLD = 0.005;
|
||||
// Pads keep a natural attack/release around the detected speech.
|
||||
const HEAD_PAD_SEC = 0.02;
|
||||
const TAIL_PAD_SEC = 0.05;
|
||||
|
||||
export const findSpeechBounds = (
|
||||
samples: Float32Array,
|
||||
sampleRate: number,
|
||||
threshold = DEFAULT_SILENCE_THRESHOLD,
|
||||
): SpeechBounds => {
|
||||
if (samples.length === 0 || sampleRate <= 0) {
|
||||
return { startSec: 0, endSec: 0 };
|
||||
}
|
||||
let first = -1;
|
||||
for (let i = 0; i < samples.length; i++) {
|
||||
if (Math.abs(samples[i]!) > threshold) {
|
||||
first = i;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (first === -1) {
|
||||
// All silence: play as-is rather than scheduling a zero-length chunk.
|
||||
return { startSec: 0, endSec: samples.length / sampleRate };
|
||||
}
|
||||
let last = first;
|
||||
for (let i = samples.length - 1; i >= first; i--) {
|
||||
if (Math.abs(samples[i]!) > threshold) {
|
||||
last = i;
|
||||
break;
|
||||
}
|
||||
}
|
||||
const startSec = Math.max(0, first / sampleRate - HEAD_PAD_SEC);
|
||||
const endSec = Math.min(samples.length / sampleRate, (last + 1) / sampleRate + TAIL_PAD_SEC);
|
||||
return { startSec, endSec };
|
||||
};
|
||||
@@ -0,0 +1,81 @@
|
||||
// WSOLA (waveform-similarity overlap-add) time-stretch for speech.
|
||||
//
|
||||
// Web Audio has no pitch-preserved playback rate (`preservesPitch` exists only
|
||||
// on media elements), so sentence buffers are stretched here before scheduling.
|
||||
// Tempo > 1 speeds speech up (shorter output); pitch is preserved because
|
||||
// whole waveform frames are re-laid at a different spacing instead of being
|
||||
// resampled.
|
||||
//
|
||||
// Parameters follow SoundTouch's speech defaults: 40ms frames, 8ms overlap,
|
||||
// +/-15ms similarity search. The search maximizes normalized cross-correlation
|
||||
// between the candidate frame head and the tail of what has been written, and
|
||||
// the overlap is linearly cross-faded, which keeps phase continuity through
|
||||
// each splice.
|
||||
|
||||
const FRAME_SEC = 0.04;
|
||||
const OVERLAP_SEC = 0.008;
|
||||
const SEEK_SEC = 0.015;
|
||||
|
||||
export const timeStretch = (
|
||||
input: Float32Array,
|
||||
sampleRate: number,
|
||||
tempo: number,
|
||||
): Float32Array => {
|
||||
const frame = Math.round(FRAME_SEC * sampleRate);
|
||||
const overlap = Math.round(OVERLAP_SEC * sampleRate);
|
||||
const seek = Math.round(SEEK_SEC * sampleRate);
|
||||
if (!(tempo > 0) || tempo === 1 || input.length < 2 * frame) {
|
||||
return input.slice();
|
||||
}
|
||||
|
||||
const flat = frame - overlap; // samples appended per iteration
|
||||
const inputAdvance = flat * tempo; // nominal input advance per iteration
|
||||
const out = new Float32Array(Math.ceil(input.length / tempo) + 2 * frame);
|
||||
|
||||
out.set(input.subarray(0, frame), 0);
|
||||
let outPos = frame;
|
||||
// Nominal input position accumulates independently of the chosen offsets so
|
||||
// alignment error never drifts over the sentence.
|
||||
let inNominal = 0;
|
||||
|
||||
for (;;) {
|
||||
inNominal += inputAdvance;
|
||||
const target = Math.round(inNominal);
|
||||
const searchMin = Math.max(0, target - seek);
|
||||
const searchMax = Math.min(input.length - frame, target + seek);
|
||||
if (searchMax < searchMin) break;
|
||||
|
||||
const refStart = outPos - overlap;
|
||||
let bestOffset = searchMin;
|
||||
let bestCorr = -Infinity;
|
||||
for (let off = searchMin; off <= searchMax; off++) {
|
||||
let dot = 0;
|
||||
let energy = 0;
|
||||
for (let i = 0; i < overlap; i++) {
|
||||
const a = out[refStart + i]!;
|
||||
const b = input[off + i]!;
|
||||
dot += a * b;
|
||||
energy += b * b;
|
||||
}
|
||||
// The epsilon guards the zero-energy window (digital silence) so the
|
||||
// correlation never divides to NaN.
|
||||
const corr = dot / Math.sqrt(energy + 1e-12);
|
||||
if (corr > bestCorr) {
|
||||
bestCorr = corr;
|
||||
bestOffset = off;
|
||||
}
|
||||
}
|
||||
|
||||
for (let i = 0; i < overlap; i++) {
|
||||
const t = i / overlap;
|
||||
out[refStart + i] = out[refStart + i]! * (1 - t) + input[bestOffset + i]! * t;
|
||||
}
|
||||
const copyStart = bestOffset + overlap;
|
||||
const copyEnd = Math.min(bestOffset + frame, input.length);
|
||||
out.set(input.subarray(copyStart, copyEnd), outPos);
|
||||
outPos += copyEnd - copyStart;
|
||||
if (copyEnd >= input.length) break;
|
||||
}
|
||||
|
||||
return out.slice(0, outPos);
|
||||
};
|
||||
@@ -0,0 +1,123 @@
|
||||
// Sentence duration bookkeeping for the TTS section timeline.
|
||||
//
|
||||
// Three tiers, best data wins:
|
||||
// 1. Measured durations (exact, from decoded+trimmed audio; provisional
|
||||
// values derive from word-boundary metadata at fetch time and never
|
||||
// overwrite a measured value).
|
||||
// 2. Per-voice chars-per-second calibration (EMA over measured sentences,
|
||||
// persisted in localStorage so a voice starts calibrated next session).
|
||||
// 3. Per-script defaults for the cold start.
|
||||
//
|
||||
// Keys normalize away punctuation and case because the spoken text (mark text
|
||||
// after SSML preprocessing) and the timeline text (raw range text) differ only
|
||||
// in punctuation rewrites; the letter/digit content is identical.
|
||||
|
||||
import { md5 } from 'js-md5';
|
||||
import { isCJKLang } from '@/utils/lang';
|
||||
import { LRUCache } from '@/utils/lru';
|
||||
|
||||
const CALIBRATION_STORAGE_KEY = 'readest-tts-voice-cps';
|
||||
const EMA_ALPHA = 0.2;
|
||||
const MIN_CALIBRATION_CHARS = 10;
|
||||
// Fixed per-utterance overhead floor: even a one-word sentence is not
|
||||
// instantaneous once Edge's attack/release around speech is counted.
|
||||
const MIN_SENTENCE_SEC = 0.3;
|
||||
|
||||
const CJK_DEFAULT_CPS = 4.5;
|
||||
const LATIN_DEFAULT_CPS = 15;
|
||||
|
||||
interface VoiceCalibration {
|
||||
cps: number;
|
||||
n: number;
|
||||
}
|
||||
|
||||
const measured = new LRUCache<string, number>(2048);
|
||||
const provisional = new Set<string>();
|
||||
// In-memory calibration mirror; localStorage is best-effort persistence.
|
||||
let calibrations: Record<string, VoiceCalibration> | null = null;
|
||||
|
||||
const normalizeText = (text: string): string =>
|
||||
text
|
||||
.toLowerCase()
|
||||
.replace(/[^\p{L}\p{N}]+/gu, ' ')
|
||||
.trim();
|
||||
|
||||
const durationKey = (voiceId: string, text: string): string =>
|
||||
md5(`${voiceId}|${normalizeText(text)}`);
|
||||
|
||||
const loadCalibrations = (): Record<string, VoiceCalibration> => {
|
||||
if (calibrations) return calibrations;
|
||||
calibrations = {};
|
||||
try {
|
||||
const raw = localStorage.getItem(CALIBRATION_STORAGE_KEY);
|
||||
if (raw) {
|
||||
const parsed = JSON.parse(raw) as Record<string, VoiceCalibration>;
|
||||
for (const [voiceId, cal] of Object.entries(parsed)) {
|
||||
if (typeof cal?.cps === 'number' && Number.isFinite(cal.cps) && cal.cps > 0) {
|
||||
calibrations[voiceId] = { cps: cal.cps, n: cal.n || 1 };
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// localStorage unavailable or corrupt: run with in-memory calibration only.
|
||||
}
|
||||
return calibrations;
|
||||
};
|
||||
|
||||
const saveCalibrations = () => {
|
||||
if (!calibrations) return;
|
||||
try {
|
||||
localStorage.setItem(CALIBRATION_STORAGE_KEY, JSON.stringify(calibrations));
|
||||
} catch {
|
||||
// Best-effort persistence only.
|
||||
}
|
||||
};
|
||||
|
||||
export const recordMeasuredDuration = (voiceId: string, text: string, seconds: number): void => {
|
||||
if (!Number.isFinite(seconds) || seconds <= 0) return;
|
||||
const key = durationKey(voiceId, text);
|
||||
measured.set(key, seconds);
|
||||
provisional.delete(key);
|
||||
};
|
||||
|
||||
// Word-boundary-derived durations are close but not canonical; keep them only
|
||||
// until a decode-time measurement lands.
|
||||
export const recordProvisionalDuration = (voiceId: string, text: string, seconds: number): void => {
|
||||
if (!Number.isFinite(seconds) || seconds <= 0) return;
|
||||
const key = durationKey(voiceId, text);
|
||||
if (measured.get(key) !== undefined && !provisional.has(key)) return;
|
||||
measured.set(key, seconds);
|
||||
provisional.add(key);
|
||||
};
|
||||
|
||||
export const getMeasuredDuration = (voiceId: string, text: string): number | undefined =>
|
||||
measured.get(durationKey(voiceId, text));
|
||||
|
||||
export const calibrateVoiceRate = (voiceId: string, text: string, seconds: number): void => {
|
||||
if (!Number.isFinite(seconds) || seconds <= 0) return;
|
||||
const chars = normalizeText(text).length;
|
||||
if (chars < MIN_CALIBRATION_CHARS) return;
|
||||
const cps = chars / seconds;
|
||||
const all = loadCalibrations();
|
||||
const existing = all[voiceId];
|
||||
if (existing) {
|
||||
existing.cps = existing.cps * (1 - EMA_ALPHA) + cps * EMA_ALPHA;
|
||||
existing.n += 1;
|
||||
} else {
|
||||
all[voiceId] = { cps, n: 1 };
|
||||
}
|
||||
saveCalibrations();
|
||||
};
|
||||
|
||||
export const defaultCharsPerSecond = (lang: string): number =>
|
||||
isCJKLang(lang) ? CJK_DEFAULT_CPS : LATIN_DEFAULT_CPS;
|
||||
|
||||
export const estimateSentenceSeconds = (text: string, lang: string, voiceId: string): number => {
|
||||
const measuredSec = getMeasuredDuration(voiceId, text);
|
||||
if (measuredSec !== undefined) return measuredSec;
|
||||
const chars = normalizeText(text).length;
|
||||
if (chars === 0) return 0;
|
||||
const calibrated = loadCalibrations()[voiceId];
|
||||
const cps = calibrated?.cps ?? defaultCharsPerSecond(lang);
|
||||
return Math.max(MIN_SENTENCE_SEC, chars / cps);
|
||||
};
|
||||
@@ -34,3 +34,18 @@ export const initDayjs = (locale: string) => {
|
||||
dayjs.locale(locale);
|
||||
dayjs.extend(relativeTime);
|
||||
};
|
||||
|
||||
// Clock-style playback time for the TTS scrubber: m:ss below one hour,
|
||||
// h:mm:ss above. Pass forceHours so both labels of a row share the format
|
||||
// chosen by the total's magnitude and the row never re-layouts when the
|
||||
// elapsed side crosses an hour.
|
||||
export const formatPlaybackTime = (seconds: number, forceHours = false): string => {
|
||||
const total = Number.isFinite(seconds) && seconds > 0 ? Math.floor(seconds) : 0;
|
||||
const hours = Math.floor(total / 3600);
|
||||
const minutes = Math.floor((total % 3600) / 60);
|
||||
const secs = total % 60;
|
||||
if (hours > 0 || forceHours) {
|
||||
return `${hours}:${String(minutes).padStart(2, '0')}:${String(secs).padStart(2, '0')}`;
|
||||
}
|
||||
return `${minutes}:${String(secs).padStart(2, '0')}`;
|
||||
};
|
||||
|
||||
+1
-1
Submodule packages/foliate-js updated: cecaef95be...6992cfef0c
Reference in New Issue
Block a user