---
name: wordlens-feature
description: "Word Lens inline-gloss feature — CFI-safe ruby technique, TTS/search isolation, gloss-data build caveat"
metadata: 
  node_type: memory
  type: project
  originSessionId: 5a24d2e1-f19b-4e29-b1f6-e5e4969e2a7b
---

Kindle-style **Word Lens**: a short native-language gloss above difficult words as you read, gated by a **CEFR vocabulary-level slider (A1–C2)** mapped to frequency-rank cutoffs in `difficulty.ts` (A1=most hints/low cutoff … C2=fewest; an approximation, NOT per-word CEFR; zh = analogy over HSK), tap-to-open the existing dictionary. **Shipped as PR #4589 (Closes #2146)** — branch `feat/word-wise` squashed to 2 commits (code, then `data(wordlens)`), rebased on origin/main, pushed. Worktree `/Users/chrox/dev/readest-feat-word-wise`. Spec + plan in `apps/readest-app/docs/superpowers/{specs,plans}/2026-06-14-word-lens*`. (Do NOT reference "jinshu" in repo docs/code — scrubbed per user request.)

**Files:** `src/services/wordlens/{types,difficulty,glossIndex,planner}.ts` (pure pipeline), `src/app/reader/utils/{wordlensRuby,wordlensSection}.ts` (DOM inject/clear + orchestration), `src/components/settings/WordLensPanel.tsx`, `scripts/build-wordlens-data.mjs`, `public/wordlens/{en-zh,zh-en}.json` + `ATTRIBUTION.md`.

**CFI-safe DOM mutation (the key technique).** Glosses render as `<ruby class="wl-gloss" cfi-skip>word<rt cfi-inert>gloss</rt></ruby>` injected into the live iframe doc. foliate `epubcfi.js` `rawChildNodes` HOISTS a `cfi-skip` wrapper's children and REMOVES a `cfi-inert` subtree, then `indexChildNodes` re-merges the now-adjacent text nodes into one chunk → CFI is byte-identical to plain text in BOTH `fromRange` and `toRange`. So highlights/bookmarks/progress are unaffected. `xcfi.ts` honors the same attrs → KOSync safe too. Gate test: `epubcfi-ruby-inline.test.ts` (the pre-existing `epubcfi-{inert,skip}.test.ts` only cover block wrappers, NOT mid-text inline ruby with text on both sides). NOTE: an early research subagent wrongly claimed `cfi-skip`/`cfi-inert` only affect tree-walk not offset summation — that's FALSE; they fully neutralize offsets. This retired the need for an overlay-rendering fallback.

**TTS/search/annotation isolation.** Gloss text must stay out of spoken text, find-in-book, and Edge-TTS word offsets. (1) foliate TTS reject filter already has `tags:['rt']` (`TTSController.ts`) → spoken text excludes gloss automatically. (2) `globalAnnotations` findTextRanges already rejects `rt`/`rp`. (3) `wordHighlight.ts` getTextSubRange walked raw `SHOW_TEXT` → added `isInertText` skip + exported `rangeTextExcludingInert` (used in `TTSController.prepareSpeakWords` instead of `range.toString()`), else Edge word offsets drift vs the gloss-free boundary words — this also fixes a latent furigana bug. (4) search `SearchBar.tsx` only rejected `rt` for ja → added `attributes:['cfi-inert']`.

**Gloss data caveat (IMPORTANT).** The committed `public/wordlens/*.json` are small CURATED STARTER sets (52 EN→中文, 40 中文→EN) so the feature works out-of-the-box. The FULL frequency-trimmed assets are a maintainer step: `node scripts/build-wordlens-data.mjs en-zh <ecdict.csv> [topN]` (ECDICT, MIT, ~160MB full; difficulty = COCA `frq` rank; EN inflections parsed from the `exchange` field) and `zh-en <cedict.txt> <hsk.json> [topN]` (CC-CEDICT CC-BY-SA + HSK; zh tokenized in-app via jieba `cutZh`). Index is lazy-fetched once from `/wordlens/<pair>.json` into an in-memory `GlossIndex`.

**Gotchas fixed in review:** slider-drag fires repeated refreshes → `refreshSectionGlosses` uses a `WeakMap<Document,number>` generation guard (latest wins on a clean doc); a `pendingWordLensDictRef` could stick `true` past early-returns and hijack the next selection → read-and-reset pattern; skip pre-paginated/fixed-layout books (ruby overflows fixed boxes); adding a `SettingsPanelType` requires updating the exhaustive `panelIcons` map in `commandRegistry.ts` AND its `react-icons/pi` test mock. Tap-to-dict = `iframeEventHandlers` hit-tests `.wl-gloss` → `wordlens-dictionary` event → `Annotator` synthesizes a selection over the base word. **STALE-CLOSURE bug (fixed post-PR):** `handleWordLensDictionary` is registered in a `useEffect(…, [bookKey])`, so it closes over the render-time `view` const — which is still `null` when the effect first runs (view mounts after bookKey). At tap time the closure's `view` stays null → `view.renderer.getContents()` undefined → content-not-found early-return → no popup, silent. Fix = read `const view = getView(bookKey)` FRESH inside the handler (mirrors the other handlers at lines ~616/709/748 that already do this). Diagnosed live via `[wwtap]` console-trace instrumentation through the whole path (gloss-hit → dispatch → handler → effect); the `content? true index N` log confirmed the fix. General rule for this file: any foliate event handler bound once with `[bookKey]` deps must call `getView(bookKey)` rather than use the closure `view`.

**R2 runtime delivery (no bundled packs).** Packs are NOT shipped in the installer: source of truth is `apps/readest-app/data/wordlens/` (committed, outside `public/`+`src/` so unbundled) holding `<pair>.json` + `manifest.json` (`{pair,source,target,file,bytes,sha256,entries}`); `scripts/sync-wordlens-r2.mjs` (`pnpm wordlens:sync`, env `WORDLENS_R2_BUCKET`) uploads to the `cdn.readest.com` R2 bucket (`WORDLENS_CDN_BASE`). Runtime: `src/services/wordlens/glossPacks.ts` = `fetchManifest`→`resolvePack(source,hint)`→`ensurePack`(sha256 verify, single-flight via in-flight Map, `allowDownload` flag, `?v=<sha8>` cache-bust on pack URL only)→`loadGlossIndex` (memo does NOT cache null → retries after download), stored in appService `'Data'` (IndexedDB on web). `getPackStatus`/`deletePack`/`listAvailableTargets` drive the UI.

**CRITICAL Tauri download gotcha (bug-class, invisible to tests):** to download a file cross-platform you MUST write to an ABSOLUTE path (`await appService.resolveFilePath(rel,'Data')`) and read/delete it with base `'None'`. The Rust `download_file`/`tauriDownload` resolves a relative `dst` against the process CWD, but `appService.readFile(rel,'Data')` resolves against AppData → silent mismatch → download "fails" → feature dead on desktop/mobile. Tests that inject a fake downloader NEVER catch this (stub `defaultDownloader`); add a `downloadViaTempFile(appService, downloadFileFn, …)` helper test with a fake AppService keyed by absolute path. Route downloads through the Rust path (not webview `fetch`) to avoid cross-origin/CORS on the webview — CSP is fine (`connect-src` whitelists `https://*.readest.com`). Web branch uses `webDownload` (fetch+stream). See [[android-nativefile-remotefile-io]] for related Tauri IO.

**Real pack generation + data stack (lightweight).** Generated 13 committed packs (~11MB total) in `data/wordlens/`: en↔中文 via ECDICT(MIT)/CC-CEDICT(CC-BY-SA)+HSK (highest quality there); es/fr/de/pt/it/ru↔en via the lightweight stack. Difficulty = **FrequencyWords** (hermitdave, OpenSubtitles/OPUS, CC-BY-SA-4.0, `word count` lines, rank=line#). Glosses = **WikDict** (`download.wikdict.com/dictionaries/sqlite/2/<pair>.sqlite3`, per-pair SQLite 6–30MB, both directions, DBnary/Wiktionary, CC-BY-SA-3.0) — query `simple_translation(written_rep, trans_list)`, trans_list is `|`-joined best-first. CLI `build-wikdict <src> <tgt> <freq.txt> <db.sqlite3> [topN]` (needs `sqlite3` CLI). TRAP: the kaikki 100MB-gz `downloads/<code>/<code>-extract.jsonl.gz` is the MONOLINGUAL Wiktionary (same-language defs, 0% English) — NOT bilingual; the English glosses live in the heavy `kaikki.org/dictionary/<Language>/...jsonl` (0.9–3GB, CC-BY-SA-4.0, the `build` CLI mode) or the 22GB all-langs dump (never needed; never shipped — shipped packs are ~0.3–2.6MB, downloaded on demand). `buildPack` `skipTop:1000` (don't gloss the commonest words) + `topN` (default 20000). `shortGloss` strips POS/`[…]`/`CL:` + 24-char cap. ja/ko/th X→en still deferred (need a segmenter). gloss data files are CC-BY-SA → ship the NOTICE in `data/wordlens/ATTRIBUTION.md` + each pack's `meta`.

**UI in Settings → Language** (not a top-level tab): `WordLensSubPanel` opened via a `NavigationRow` in `LangPanel`, mirroring the `CustomDictionaries` sub-page swap (`useKeyDownActions` back/Esc + early-return render). Hint-language `SettingsSelect` = `getLangOptions(TRANSLATED_LANGS)` base-code-filtered against manifest targets (targets are base codes `zh`; TRANSLATED_LANGS has `zh-CN`/`pt-BR` → match on base); value `wordLensHintLang` ('' = auto = app UI lang via `getLocale()`). Global `ReadSettings.wordLensAutoDownload` (default true) gates reader auto-download; `isMetered()` (`src/utils/network.ts`, Network Information API, absent on iOS/Tauri → unmetered). Generalized: `WordLensSourceLang=string`; `difficulty.ts` FREQUENCY cutoffs for all non-zh + HSK-scale (`level×3000`) for zh; `canTokenizeSource` blocks ja/ko/th (tier 3, need a segmenter). Spec: `docs/superpowers/specs/2026-06-14-wordlens-r2-delivery.md`.

**Hint quality (runtime, best-effort over shipped packs — no regen).** `src/services/wordlens/gloss.ts`: `cleanGloss` strips a leading POS tag (`/^\s*(?:[a-zA-Z]{1,6}\.\s*)+/` — the `{1,6}` is so 6-letter `interj.` is caught, e.g. "Ahem" → 呃哼 not "interj. 呃哼") and keeps ONLY the first sense (split on `[,，、;；/]`), so en-zh hints stay short; applied in `planGlosses` when building each occurrence. **Derivational reduction (English source only):** a would-be-glossed word inherits a known base form's LOWER rank when the base exists in the pack AND their glosses overlap, so transparent derivations drop below the cutoff and aren't hinted (lazily⇐lazy, shyly⇐shy via shared 怯, sorrowful⇐sorrow, downwards⇐downward, inwards⇐inward). `baseFormCandidates` over-generates via reverse-suffix rules (-ly/-ily→y/-ful/-ward(s)/-ness/-less); `glossesShareMeaning` validates (shared Han char after stripping 的/地 particles; Latin = shared word≥3). Drifted forms keep their own rank because their gloss does NOT overlap the base (hardly≠hard, lately≠late, barely≠bare) — and those are sub-B1-common anyway so unaffected at higher levels. Done in the PLANNER (which knows `sourceLang`), GlossIndex stays pure. The matters-only-when-would-be-glossed gate (compute reduction only after the direct rank passes `isDifficult`) keeps it cheap. **NOT in the rename PR #4633** — landed as a follow-up; the `wl-gloss` class rename (was `ww-gloss`, the 'ww' shorthand #4633 missed) rode along.

Related: [[edge-tts-word-highlighting-4017]], [[footnote-aside-namespace-order-4438]], [[empty-start-cfi-sync]], [[android-nativefile-remotefile-io]].
