feat(android): add D-pad navigation for Android TV remote controller (#3745)

* feat(android): add D-pad navigation for Android TV remote controller

Add basic D-pad/arrow key navigation support for Android TV Bluetooth
remote controllers, covering the full flow: library grid browsing,
reader page turning, toolbar interaction, and TTS toggle.

Library:
- Custom spatial navigation hook for grid D-pad navigation
- Arrow keys move between BookshelfItem elements with auto column detection
- ArrowDown from header enters the bookshelf grid

Reader:
- Enter key toggles header/footer toolbar visibility
- Left/Right navigate between toolbar buttons, Up/Down moves between
  header and footer bars
- Back button dismisses toolbar (with blur) before sidebar/library
- Focus-probe technique for reliable button visibility detection
  across fixed/hidden containers

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: only auto-focus toolbar buttons on keyboard activation, not mouse hover

Track pointer activity to distinguish keyboard vs mouse toolbar activation.
When the toolbar appears due to mouse hover, skip auto-focus to prevent
unwanted focus outlines on desktop.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: track keyboard events instead of pointer events for auto-focus guard

Pointer events from iframes don't bubble to the main document, so
tracking pointermove/pointerdown missed hover interactions over book
content. Track keydown instead — auto-focus only when a recent keyboard
event (Enter key) triggered the toolbar, not mouse hover.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Huang Xin
2026-04-05 01:50:34 +08:00
committed by GitHub
parent 9595aa56e0
commit f361698e05
11 changed files with 472 additions and 2 deletions
@@ -16,6 +16,9 @@
- `src/app/reader/components/FoliateViewer.tsx` - Reader view orchestration
- `src/app/reader/components/annotator/Annotator.tsx` - Annotation lifecycle
## Feature Notes
- [D-pad Navigation](dpad-navigation.md) — Android TV remote / keyboard arrow navigation design, key files, and pitfalls
## Architecture Notes
- foliate-js is a git submodule at `packages/foliate-js/`
- Multiview paginator: loads adjacent sections in background, multiple View/Overlayer instances per book
@@ -0,0 +1,40 @@
---
name: D-pad Navigation Design
description: Android TV / Bluetooth remote D-pad navigation architecture, key files, and pitfalls encountered during implementation
type: project
---
## D-pad Navigation Architecture
D-pad support enables Bluetooth remote controller navigation on Android TV (and keyboard arrow navigation on desktop).
### Key Files
- `src/app/reader/hooks/useSpatialNavigation.ts` — Reader toolbar D-pad navigation. Left/Right navigates between buttons, Up/Down moves between header↔footer. Auto-focuses first button on show. Uses focus-probe technique for visibility detection.
- `src/app/library/hooks/useSpatialNavigation.ts` — Library grid D-pad navigation. Arrow keys move between BookshelfItem elements. ArrowDown from outside bookshelf (e.g. header) enters the grid via window-level listener.
- `src/helpers/shortcuts.ts``onToggleToolbar` (Enter key) toggles reader toolbar visibility.
- `src/app/reader/hooks/useBookShortcuts.ts``toggleToolbar` handler shows/hides header+footer bars. Skips when a `<button>` is focused (lets native click fire).
- `src/__tests__/hooks/useSpatialNavigation.test.tsx` — Unit tests for reader toolbar navigation.
### Design Decisions
- **No third-party library**: Tried `@noriginmedia/norigin-spatial-navigation` but it failed due to init timing issues (React child effects run before parent effects) and conflicts with the existing `useShortcuts` system. Custom solution is simpler and more reliable.
- **Two `useSpatialNavigation` hooks**: Same name in different directories — library version handles grid navigation, reader version handles toolbar button navigation. Different navigation patterns but same concept.
- **Platform-agnostic hooks**: Both `useSpatialNavigation` hooks work on all platforms, not just Android.
- **Focus-probe for visibility**: `offsetParent` is unreliable for detecting visible buttons (returns null inside `position: fixed` containers on mobile). Instead, try `btn.focus()` and check if `document.activeElement === btn` — this correctly handles all hiding methods (display:none, visibility:hidden, fixed positioning).
### Pitfalls
1. **WebView spatial navigation conflict**: Android WebView has built-in spatial navigation that intercepts D-pad arrow keys and moves DOM focus between `tabIndex>=0` elements. Added `tabIndex={-1}` to non-interactive overlay elements (HeaderBar trigger, ProgressBar, FooterBar trigger, SectionInfo) to prevent focus theft.
2. **`eventDispatcher.dispatchSync` short-circuits**: When multiple handlers are registered for `native-key-down`, the first handler returning `true` stops propagation. The FooterBar's Back handler fires before the Reader's. Both must independently call `blur()` — can't rely on the Reader's handler running.
3. **Must blur on toolbar dismiss**: When Back/Escape dismisses the toolbar, the focused button must be blurred. Otherwise `document.activeElement` remains a hidden button, and `toggleToolbar` skips Enter when `activeElement.tagName === 'BUTTON'`. Blur is called in FooterBar's handleKeyDown (for Back and Escape) and in Reader's handleKeyDown.
4. **Arrow key trapping must use `stopPropagation`**: Without it, arrow keys bubble to `window` where `useShortcuts` handles them as page turns. The toolbar keydown handler on the container div calls `e.stopPropagation()` + `e.preventDefault()` to prevent this.
5. **Library grid needs window-level listener**: The bookshelf container keydown handler only fires when focus is inside it. A separate `window` keydown listener handles ArrowDown from the header into the grid (when focus is outside the container).
6. **Auto-focus race on toolbar show**: Both header and footer bars auto-focus their first button when `isVisible` becomes true simultaneously. The last effect to run wins. This is acceptable — user can navigate between them with Up/Down.
7. **`offsetParent` null in fixed containers**: On mobile, `.footer-bar` uses `position: fixed`. All child buttons have `offsetParent === null`, making `offsetParent`-based visibility checks useless. The focus-probe approach (try focus, check activeElement) is the reliable alternative.
@@ -0,0 +1,188 @@
import { cleanup, render, act } from '@testing-library/react';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import React, { useRef } from 'react';
import {
useSpatialNavigation,
_resetLastKeyboardTime,
} from '@/app/reader/hooks/useSpatialNavigation';
function TestToolbar({ isVisible }: { isVisible: boolean }) {
const ref = useRef<HTMLDivElement>(null);
useSpatialNavigation(ref, isVisible);
return (
<div ref={ref} data-testid='toolbar'>
<button data-testid='btn-1'>Button 1</button>
<button data-testid='btn-2'>Button 2</button>
<button data-testid='btn-3'>Button 3</button>
<button disabled data-testid='btn-disabled'>
Disabled
</button>
</div>
);
}
function TestHeaderFooter({ isVisible }: { isVisible: boolean }) {
const headerRef = useRef<HTMLDivElement>(null);
const footerRef = useRef<HTMLDivElement>(null);
useSpatialNavigation(headerRef, isVisible);
useSpatialNavigation(footerRef, isVisible);
return (
<>
<div ref={headerRef} className='header-bar' data-testid='header'>
<button data-testid='header-btn-1'>H1</button>
<button data-testid='header-btn-2'>H2</button>
</div>
<div ref={footerRef} className='footer-bar' data-testid='footer'>
<button data-testid='footer-btn-1'>F1</button>
<button data-testid='footer-btn-2'>F2</button>
</div>
</>
);
}
function pressKey(element: HTMLElement, key: string) {
const event = new KeyboardEvent('keydown', {
key,
bubbles: true,
cancelable: true,
});
const spy = vi.spyOn(event, 'stopPropagation');
element.dispatchEvent(event);
return { event, stopPropagationCalled: spy.mock.calls.length > 0 };
}
function simulateKeyboardActivation() {
document.dispatchEvent(new KeyboardEvent('keydown', { key: 'Enter', bubbles: true }));
}
async function waitForAutoFocus() {
await act(() => new Promise((r) => setTimeout(r, 150)));
}
describe('useToolbarKeyNavigation', () => {
beforeEach(() => {
_resetLastKeyboardTime();
});
afterEach(() => cleanup());
describe('auto-focus', () => {
it('focuses first enabled button when keyboard-activated', async () => {
simulateKeyboardActivation();
render(<TestToolbar isVisible={true} />);
await waitForAutoFocus();
expect(document.activeElement).toBe(document.querySelector('[data-testid="btn-1"]'));
});
it('does not auto-focus when not visible', async () => {
simulateKeyboardActivation();
render(<TestToolbar isVisible={false} />);
await waitForAutoFocus();
expect(document.activeElement).not.toBe(document.querySelector('[data-testid="btn-1"]'));
});
it('does not auto-focus when mouse-activated (no recent keyboard)', async () => {
// No simulateKeyboardActivation() — simulates mouse hover activation
render(<TestToolbar isVisible={true} />);
await waitForAutoFocus();
expect(document.activeElement).not.toBe(document.querySelector('[data-testid="btn-1"]'));
});
});
describe('left/right navigation', () => {
it('ArrowRight moves focus to the next button', async () => {
simulateKeyboardActivation();
render(<TestToolbar isVisible={true} />);
await waitForAutoFocus();
const btn1 = document.querySelector<HTMLElement>('[data-testid="btn-1"]')!;
const btn2 = document.querySelector<HTMLElement>('[data-testid="btn-2"]')!;
expect(document.activeElement).toBe(btn1);
pressKey(btn1, 'ArrowRight');
expect(document.activeElement).toBe(btn2);
});
it('ArrowLeft moves focus to the previous button', async () => {
simulateKeyboardActivation();
render(<TestToolbar isVisible={true} />);
await waitForAutoFocus();
const btn1 = document.querySelector<HTMLElement>('[data-testid="btn-1"]')!;
const btn2 = document.querySelector<HTMLElement>('[data-testid="btn-2"]')!;
btn2.focus();
pressKey(btn2, 'ArrowLeft');
expect(document.activeElement).toBe(btn1);
});
it('ArrowRight does not go past the last enabled button', async () => {
simulateKeyboardActivation();
render(<TestToolbar isVisible={true} />);
await waitForAutoFocus();
const btn3 = document.querySelector<HTMLElement>('[data-testid="btn-3"]')!;
btn3.focus();
pressKey(btn3, 'ArrowRight');
expect(document.activeElement).toBe(btn3);
});
it('ArrowLeft does not go before the first button', async () => {
simulateKeyboardActivation();
render(<TestToolbar isVisible={true} />);
await waitForAutoFocus();
const btn1 = document.querySelector<HTMLElement>('[data-testid="btn-1"]')!;
pressKey(btn1, 'ArrowLeft');
expect(document.activeElement).toBe(btn1);
});
it('ArrowRight calls stopPropagation', async () => {
simulateKeyboardActivation();
render(<TestToolbar isVisible={true} />);
await waitForAutoFocus();
const btn1 = document.querySelector<HTMLElement>('[data-testid="btn-1"]')!;
const { stopPropagationCalled } = pressKey(btn1, 'ArrowRight');
expect(stopPropagationCalled).toBe(true);
});
});
describe('up/down navigation between header and footer', () => {
it('ArrowDown from header focuses first footer button', async () => {
render(<TestHeaderFooter isVisible={true} />);
await waitForAutoFocus();
const headerBtn = document.querySelector<HTMLElement>('[data-testid="header-btn-1"]')!;
const footerBtn1 = document.querySelector<HTMLElement>('[data-testid="footer-btn-1"]')!;
headerBtn.focus();
pressKey(headerBtn, 'ArrowDown');
expect(document.activeElement).toBe(footerBtn1);
});
it('ArrowUp from footer focuses first header button', async () => {
render(<TestHeaderFooter isVisible={true} />);
await waitForAutoFocus();
const headerBtn1 = document.querySelector<HTMLElement>('[data-testid="header-btn-1"]')!;
const footerBtn = document.querySelector<HTMLElement>('[data-testid="footer-btn-1"]')!;
footerBtn.focus();
pressKey(footerBtn, 'ArrowUp');
expect(document.activeElement).toBe(headerBtn1);
});
it('ArrowDown from header calls stopPropagation', async () => {
render(<TestHeaderFooter isVisible={true} />);
await waitForAutoFocus();
const headerBtn = document.querySelector<HTMLElement>('[data-testid="header-btn-1"]')!;
headerBtn.focus();
const { stopPropagationCalled } = pressKey(headerBtn, 'ArrowDown');
expect(stopPropagationCalled).toBe(true);
});
});
});
@@ -32,6 +32,7 @@ import {
} from '../utils/libraryUtils';
import { eventDispatcher } from '@/utils/event';
import { useSpatialNavigation } from '../hooks/useSpatialNavigation';
import Alert from '@/components/Alert';
import Spinner from '@/components/Spinner';
import ModalPortal from '@/components/ModalPortal';
@@ -101,6 +102,7 @@ const Bookshelf: React.FC<BookshelfProps> = ({
const isImportingBook = useRef(false);
const iconSize15 = useResponsiveSize(15);
const autofocusRef = useAutoFocus<HTMLDivElement>();
useSpatialNavigation(autofocusRef);
const { setCurrentBookshelf, setLibrary, updateBooks } = useLibraryStore();
const { setSelectedBooks, getSelectedBooks, toggleSelectedBook } = useLibraryStore();
@@ -0,0 +1,95 @@
import { useEffect, useCallback } from 'react';
const ARROW_KEYS = new Set(['ArrowLeft', 'ArrowRight', 'ArrowUp', 'ArrowDown']);
function getGridColumnCount(items: HTMLElement[]): number {
if (items.length < 2) return 1;
const firstTop = items[0]!.getBoundingClientRect().top;
for (let i = 1; i < items.length; i++) {
if (items[i]!.getBoundingClientRect().top > firstTop + 5) return i;
}
return items.length;
}
function getGridItems(container: HTMLElement): HTMLElement[] {
return Array.from(container.querySelectorAll<HTMLElement>('[role="button"][tabindex="0"]'));
}
export function useSpatialNavigation(containerRef: React.RefObject<HTMLElement | null>) {
// Grid navigation within the bookshelf
const handleGridKeyDown = useCallback(
(e: KeyboardEvent) => {
if (!ARROW_KEYS.has(e.key)) return;
const container = containerRef.current;
if (!container) return;
const items = getGridItems(container);
if (items.length === 0) return;
const currentIndex = items.indexOf(document.activeElement as HTMLElement);
if (currentIndex === -1) {
items[0]?.focus();
e.preventDefault();
return;
}
const cols = getGridColumnCount(items);
let targetIndex = currentIndex;
switch (e.key) {
case 'ArrowRight':
targetIndex = currentIndex + 1;
break;
case 'ArrowLeft':
targetIndex = currentIndex - 1;
break;
case 'ArrowDown':
targetIndex = currentIndex + cols;
break;
case 'ArrowUp':
targetIndex = currentIndex - cols;
break;
}
if (targetIndex >= 0 && targetIndex < items.length && targetIndex !== currentIndex) {
items[targetIndex]?.focus();
items[targetIndex]?.scrollIntoView({ block: 'nearest' });
e.preventDefault();
}
},
[containerRef],
);
// Handle focus transition from outside the bookshelf (e.g. header) into the grid
const handleWindowKeyDown = useCallback(
(e: KeyboardEvent) => {
if (e.key !== 'ArrowDown') return;
const container = containerRef.current;
if (!container) return;
// Only handle when focus is outside the bookshelf
if (container.contains(document.activeElement)) return;
const items = getGridItems(container);
if (items.length === 0) return;
items[0]?.focus();
items[0]?.scrollIntoView({ block: 'nearest' });
e.preventDefault();
},
[containerRef],
);
useEffect(() => {
const container = containerRef.current;
if (!container) return;
container.addEventListener('keydown', handleGridKeyDown);
window.addEventListener('keydown', handleWindowKeyDown);
return () => {
container.removeEventListener('keydown', handleGridKeyDown);
window.removeEventListener('keydown', handleWindowKeyDown);
};
}, [handleGridKeyDown, handleWindowKeyDown, containerRef]);
}
@@ -13,6 +13,7 @@ import { useSettingsStore } from '@/store/settingsStore';
import { useTrafficLightStore } from '@/store/trafficLightStore';
import { useTrafficLight } from '@/hooks/useTrafficLight';
import { useResponsiveSize } from '@/hooks/useResponsiveSize';
import { useSpatialNavigation } from '@/app/reader/hooks/useSpatialNavigation';
import { getHighlightColorHex } from '../utils/annotatorUtil';
import { annotationToolQuickActions } from './annotator/AnnotationTools';
import { AnnotationToolType } from '@/types/annotator';
@@ -133,6 +134,8 @@ const HeaderBar: React.FC<HeaderBarProps> = ({
const isHeaderCompact = headerWidth > 0 && headerWidth < 350;
const insets = window.innerWidth < 640 ? screenInsets : gridInsets;
const isHeaderVisible = hoveredBookKey === bookKey || isDropdownOpen;
useSpatialNavigation(headerRef, isHeaderVisible);
const trafficLightInHeader =
appService?.hasTrafficLight && !trafficLightInFullscreen && !isSideBarVisible && isTopLeft;
@@ -106,7 +106,11 @@ const Reader: React.FC<{ ids?: string }> = ({ ids }) => {
const handleKeyDown = (event: CustomEvent) => {
if (event.detail.keyName === 'Back') {
if (getIsSideBarVisible() && !isSideBarPinned) {
const { hoveredBookKey, setHoveredBookKey } = useReaderStore.getState();
if (hoveredBookKey) {
setHoveredBookKey('');
(document.activeElement as HTMLElement)?.blur();
} else if (getIsSideBarVisible() && !isSideBarPinned) {
setSideBarVisible(false);
} else if (getIsNotebookVisible() && !isNotebookPinned) {
setNotebookVisible(false);
@@ -1,6 +1,7 @@
import clsx from 'clsx';
import React, { useState, useCallback, useMemo, useEffect } from 'react';
import React, { useState, useCallback, useMemo, useEffect, useRef } from 'react';
import { useEnv } from '@/context/EnvContext';
import { useSpatialNavigation } from '@/app/reader/hooks/useSpatialNavigation';
import { useReaderStore } from '@/store/readerStore';
import { useSidebarStore } from '@/store/sidebarStore';
import { useBookDataStore } from '@/store/bookDataStore';
@@ -159,11 +160,13 @@ const FooterBar: React.FC<FooterBarProps> = ({
if (event instanceof CustomEvent) {
if (event.detail.keyName === 'Back') {
setHoveredBookKey('');
(document.activeElement as HTMLElement)?.blur();
return true;
}
} else {
if (event.key === 'Escape') {
setHoveredBookKey('');
(document.activeElement as HTMLElement)?.blur();
}
event.stopPropagation();
}
@@ -188,6 +191,9 @@ const FooterBar: React.FC<FooterBarProps> = ({
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [hoveredBookKey]);
const footerBarRef = useRef<HTMLDivElement>(null);
useSpatialNavigation(footerBarRef, isVisible);
const commonProps: FooterBarChildProps = {
bookKey,
gridInsets,
@@ -236,6 +242,7 @@ const FooterBar: React.FC<FooterBarProps> = ({
{/* Main footer container */}
<div
ref={footerBarRef}
role='contentinfo'
aria-label={_('Footer Bar')}
className={containerClasses}
@@ -256,6 +256,15 @@ const useBookShortcuts = ({ sideBarBookKey, bookKeys }: UseBookShortcutsProps) =
applyZoomLevel(100);
};
const toggleToolbar = () => {
if (!sideBarBookKey) return;
// Don't intercept Enter when a button is focused (let native click fire)
const active = document.activeElement;
if (active && active.tagName === 'BUTTON') return;
const { hoveredBookKey, setHoveredBookKey } = useReaderStore.getState();
setHoveredBookKey(hoveredBookKey === sideBarBookKey ? '' : sideBarBookKey);
};
const toggleTTS = () => {
if (!sideBarBookKey) return;
const bookKey = sideBarBookKey;
@@ -337,6 +346,7 @@ const useBookShortcuts = ({ sideBarBookKey, bookKeys }: UseBookShortcutsProps) =
onToggleScrollMode: toggleScrollMode,
onToggleBookmark: toggleBookmark,
onToggleParagraphMode: toggleParagraphMode,
onToggleToolbar: toggleToolbar,
onOpenFontLayoutSettings: () => setSettingsDialogOpen(true),
onShowSearchBar: showSearchBar,
onToggleFullscreen: toggleFullscreen,
@@ -0,0 +1,113 @@
import { useEffect, useRef } from 'react';
// Track last keyboard activity to distinguish keyboard vs mouse activation.
// Pointer events from iframes don't bubble to the main document, but keyboard
// events always reach here via useShortcuts on window.
let lastKeyboardTime = 0;
/** @internal Reset for testing only */
export function _resetLastKeyboardTime() {
lastKeyboardTime = 0;
}
if (typeof document !== 'undefined') {
document.addEventListener(
'keydown',
() => {
lastKeyboardTime = Date.now();
},
true,
);
}
function getButtons(container: HTMLElement): HTMLButtonElement[] {
return Array.from(container.querySelectorAll<HTMLButtonElement>('button:not([disabled])'));
}
function getFocusableButtons(container: HTMLElement): HTMLButtonElement[] {
// Try each button to see if it can actually receive focus.
// This correctly handles all hiding methods (display:none, visibility:hidden,
// fixed positioning where offsetParent is null, etc.)
const prev = document.activeElement as HTMLElement | null;
const focusable: HTMLButtonElement[] = [];
for (const btn of getButtons(container)) {
btn.focus();
if (document.activeElement === btn) {
focusable.push(btn);
}
}
prev?.focus();
return focusable;
}
function focusFirstButton(container: HTMLElement) {
for (const btn of getButtons(container)) {
btn.focus();
if (document.activeElement === btn) return;
}
}
/**
* Arrow key navigation for toolbar containers (header bar, footer bar).
* Left/Right navigate between buttons within the toolbar.
* Up/Down move focus between the header bar and footer bar.
* Auto-focuses the first button when the toolbar becomes visible.
*/
export function useSpatialNavigation(
containerRef: React.RefObject<HTMLElement | null>,
isVisible: boolean,
) {
// Auto-focus first button when toolbar becomes visible via keyboard (not mouse hover)
const wasKeyboardActivated = useRef(false);
useEffect(() => {
if (isVisible) {
// If recent keyboard activity, this was keyboard-activated (Enter key)
wasKeyboardActivated.current = Date.now() - lastKeyboardTime < 200;
}
}, [isVisible]);
useEffect(() => {
if (!isVisible || !wasKeyboardActivated.current) return;
const container = containerRef.current;
if (!container) return;
const timer = setTimeout(() => focusFirstButton(container), 100);
return () => clearTimeout(timer);
}, [isVisible, containerRef]);
useEffect(() => {
if (!isVisible) return;
const container = containerRef.current;
if (!container) return;
const handleKeyDown = (e: KeyboardEvent) => {
if (e.key === 'ArrowLeft' || e.key === 'ArrowRight') {
const buttons = getFocusableButtons(container);
if (buttons.length === 0) return;
const currentIndex = buttons.indexOf(document.activeElement as HTMLButtonElement);
if (currentIndex === -1) return;
const targetIndex = e.key === 'ArrowRight' ? currentIndex + 1 : currentIndex - 1;
if (targetIndex >= 0 && targetIndex < buttons.length) {
buttons[targetIndex]?.focus();
}
e.stopPropagation();
e.preventDefault();
} else if (e.key === 'ArrowDown' || e.key === 'ArrowUp') {
const target =
e.key === 'ArrowDown'
? document.querySelector<HTMLElement>('.footer-bar')
: document.querySelector<HTMLElement>('.header-bar');
if (target) {
focusFirstButton(target);
e.stopPropagation();
e.preventDefault();
}
}
};
container.addEventListener('keydown', handleKeyDown);
return () => container.removeEventListener('keydown', handleKeyDown);
}, [isVisible, containerRef]);
}
@@ -78,6 +78,11 @@ const DEFAULT_SHORTCUTS = {
description: _('Toggle Paragraph Mode'),
section: 'General',
},
onToggleToolbar: {
keys: ['Enter'],
description: _('Toggle Toolbar'),
section: 'General',
},
onHighlightSelection: {
keys: ['ctrl+h', 'cmd+h'],
description: _('Highlight Selection'),