forked from akai/readest
Huang Xin
a1046f5684
* feat(db): add reedy schema migration with Tantivy FTS + lazy embeddings Registers a new `reedy` migration set bound to reedy.db. Creates reedy_book_meta + reedy_book_chunks with a Tantivy FTS index on chunks.text (ngram tokenizer) and a per-book position index used by BookRetriever. The vector embeddings table is intentionally NOT created here — the indexer creates it lazily on first index so the vector32(<dim>) column matches the active embedding model. Tests cover the migration applies cleanly, is idempotent, and that the FTS index is queryable. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(reedy): retrieval primitives — DB, chunker, indexer, retriever, lookupPassage tool Wires the MVP retrieval pipeline behind reedy.db, all under src/services/reedy/ and with no integration into the existing AI module yet (Phase 1B will do that). - ReedyDb wrapper over DatabaseService: book-meta CRUD, lazy embeddings table at the active model's dim, bulk chunk + embedding writes via batch(), hybridSearch (brute-force cosine + Tantivy FTS + reciprocal-rank fusion with 3× per-path over-fetch), per-book and global wipe. Internal write queue serializes batch() calls so Turso's single-writer transaction guard doesn't trip when BookIndexer runs across books in parallel. - CfiChunker: TreeWalker over the section's DOM, ~maxChunkSize windows with paragraph > sentence > word break-points, full epubcfi(/6/N!/…) anchors, round-trip verified via CFI.toRange before each chunk lands. - BookIndexer: per-book mutex, lazy embeddings-table creation, model.batchSize embedding batches with dim assertion, terminal status transitions (indexed | empty_index | failed). Re-indexing clears prior chunks via a new ReedyDb.clearBookChunks helper. - BookRetriever: status-typed results (ok | not_indexed | empty_index | stale_index | degraded). Embedding has a 5s wall-clock budget; on timeout it falls through to FTS-only with status=degraded. - lookupPassage Vercel ai-SDK tool: Zod-validated query/topK, per-turn composite-key dedupe, parallel-call serialization, 10s per-turn budget, 6000-char result clamp, status-with-hint passthrough, and a separate serializeForModel that wraps each passage in <retrieved trust="untrusted"> with XML-escaped content so book text cannot escape the envelope. 59 unit tests; pnpm test + pnpm lint clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>