Huang Xin
4908245042
* feat(reader): Word Wise — inline native-language vocabulary hints Kindle-style Word Wise: a short native-language gloss renders above difficult words as you read (always-on ruby), gated by a CEFR vocabulary-level slider (A1–C2); tapping a glossed word opens the existing dictionary. - Pipeline: CEFR→frequency-rank difficulty, inflection-aware gloss index, pure offset-aware planner (EN regex + jieba for CJK). - Rendering: <ruby cfi-skip>…<rt cfi-inert> injected per occurrence — CFI-transparent (verified), so highlights/bookmarks/progress are unaffected; kept out of TTS word offsets and find-in-book. - Delivery: gloss packs are version-controlled in data/wordwise/, mirrored to R2, and downloaded on demand into local storage (sha-verified, single-flight) when enabled. - Settings: a Word Wise sub-page under Settings → Language (enable, level, hint language, per-pack download/manage, auto-download toggle). - Build tooling: scripts/build-wordwise-data.mjs (ECDICT / CC-CEDICT+HSK / WikDict + FrequencyWords, with lemmatization) and scripts/sync-wordwise-r2.mjs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * data(wordwise): bundled gloss packs + manifest + attribution 13 frequency-trimmed gloss packs (en↔中文 + es/fr/de/pt/it/ru↔en, ~19 MB) generated by build-wordwise-data.mjs from ECDICT (MIT), CC-CEDICT + HSK, and WikDict + FrequencyWords (CC-BY-SA). Source of truth, mirrored to the CDN via `pnpm wordwise:sync`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2.8 KiB
Word Wise data attributions
Note: The committed
en-zh.json(top 30,000 by COCA frequency) andzh-en.json(top 12,000, HSK-ranked) are frequency-trimmed derivatives generated from the datasets below viascripts/build-wordwise-data.mjs. The full source corpora (ECDICT ~66 MB, CC-CEDICT) are not committed — regenerate with the commands below. Glosses are cleaned (POS tags,[…]annotations, and CC-CEDICTCL:clauses stripped) and capped to the first 1–2 short senses.
The en↔中文 packs use dedicated dictionaries (highest quality there); all other pairs (es/fr/de/pt/it/ru ↔ en) use the lightweight WikDict + FrequencyWords stack.
Sources
- ECDICT (English→中文 glosses, frequency ranks, inflections) — MIT License. https://github.com/skywind3000/ECDICT
- CC-CEDICT (中文→English glosses) — CC-BY-SA 4.0. https://cc-cedict.org/
- HSK vocabulary levels (中文 difficulty ranking) — open dataset. https://github.com/drkameleon/complete-hsk-vocabulary
- WikDict (bilingual glosses for es/fr/de/pt/it/ru ↔ en) — CC-BY-SA 3.0, derived from DBnary / Wiktionary. https://www.wikdict.com/
- FrequencyWords (difficulty ranking; word-frequency lists from OpenSubtitles/OPUS) — CC-BY-SA 4.0. https://github.com/hermitdave/FrequencyWords
- lemmatization-lists (form→lemma mappings used to lemmatize non-English source
words, e.g. Spanish
corriendo→correr) — compiled from Wiktionary + open lemma data. https://github.com/michmech/lemmatization-lists (English form→lemma comes from ECDICT'sexchangefield instead.)
The committed *.json packs are frequency-trimmed derivatives generated by
scripts/build-wordwise-data.mjs. Redistribution complies with the above licenses; the
CC-BY-SA packs (everything except en-zh.json) carry their license + attribution in each
pack's meta. Per-pack sizes: 0.3–2.6 MB.
Regenerating the assets (maintainer step)
# English → 中文 (ECDICT csv) and 中文 → English (CC-CEDICT + HSK)
node scripts/build-wordwise-data.mjs en-zh /path/to/ecdict.csv 30000
node scripts/build-wordwise-data.mjs zh-en /path/to/cedict.txt /path/to/hsk.json 12000
# Any other pair (one side English) from WikDict + FrequencyWords. Needs the `sqlite3`
# CLI. Download <pair>.sqlite3 from https://download.wikdict.com/dictionaries/sqlite/2/
# and <src>_50k.txt from FrequencyWords/content/2018/<src>/.
node scripts/build-wordwise-data.mjs build-wikdict es en es_50k.txt es-en.sqlite3 20000
node scripts/build-wordwise-data.mjs build-wikdict en es en_50k.txt en-es.sqlite3 20000
# (repeat for fr/de/pt/it/ru ↔ en)
# For maximum coverage, the kaikki Wiktionary dump can be used instead of WikDict:
node scripts/build-wordwise-data.mjs build es en es_50k.txt /path/to/es-extract.jsonl 20000