* feat(db): add reedy schema migration with Tantivy FTS + lazy embeddings
Registers a new `reedy` migration set bound to reedy.db. Creates
reedy_book_meta + reedy_book_chunks with a Tantivy FTS index on
chunks.text (ngram tokenizer) and a per-book position index used by
BookRetriever. The vector embeddings table is intentionally NOT created
here — the indexer creates it lazily on first index so the vector32(<dim>)
column matches the active embedding model. Tests cover the migration
applies cleanly, is idempotent, and that the FTS index is queryable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(reedy): retrieval primitives — DB, chunker, indexer, retriever, lookupPassage tool
Wires the MVP retrieval pipeline behind reedy.db, all under src/services/reedy/
and with no integration into the existing AI module yet (Phase 1B will do that).
- ReedyDb wrapper over DatabaseService: book-meta CRUD, lazy embeddings
table at the active model's dim, bulk chunk + embedding writes via batch(),
hybridSearch (brute-force cosine + Tantivy FTS + reciprocal-rank fusion
with 3× per-path over-fetch), per-book and global wipe. Internal write
queue serializes batch() calls so Turso's single-writer transaction guard
doesn't trip when BookIndexer runs across books in parallel.
- CfiChunker: TreeWalker over the section's DOM, ~maxChunkSize windows with
paragraph > sentence > word break-points, full epubcfi(/6/N!/…) anchors,
round-trip verified via CFI.toRange before each chunk lands.
- BookIndexer: per-book mutex, lazy embeddings-table creation, model.batchSize
embedding batches with dim assertion, terminal status transitions
(indexed | empty_index | failed). Re-indexing clears prior chunks via a new
ReedyDb.clearBookChunks helper.
- BookRetriever: status-typed results (ok | not_indexed | empty_index |
stale_index | degraded). Embedding has a 5s wall-clock budget; on timeout
it falls through to FTS-only with status=degraded.
- lookupPassage Vercel ai-SDK tool: Zod-validated query/topK, per-turn
composite-key dedupe, parallel-call serialization, 10s per-turn budget,
6000-char result clamp, status-with-hint passthrough, and a separate
serializeForModel that wraps each passage in <retrieved trust="untrusted">
with XML-escaped content so book text cannot escape the envelope.
59 unit tests; pnpm test + pnpm lint clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(db): forward DatabaseOpts to tauri-plugin-turso
NativeDatabaseService.open ignored its opts parameter, dropping any
experimental feature flags (e.g. 'index_method' needed for FTS / vector
indexes) and any encryption config before they could reach the Tauri
plugin. Translate DatabaseOpts to the plugin's LoadOptions shape and
forward as the single argument Database.load accepts. Skip translation
when no relevant opts are set so the existing path-string call shape is
preserved for callers without experimental/encryption needs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(db): cover Turso vector primitives + add benchmark harness
Verifies the Turso functions Reedy retrieval depends on, with a
brute-force per-book kNN test that runs against every DatabaseService
backend (node, native, WASM):
SELECT vector_distance_cos(embedding, vector32(?)) AS d
FROM book_chunks
WHERE book_hash = ?
ORDER BY d ASC LIMIT k
This is the path Turso's own founder recommended in
tursodatabase/turso#3778 ("First, focus on efficient SIMD-accelerated
brute-force search") and what shipped at commit 1aba105df4f. Native
vector index modules don't exist in this engine: `libsql_vector_idx`,
`vector_top_k`, and `USING vector/hnsw/diskann/ivfflat` all parse-error
against @tursodatabase/database@0.6.0-pre.28 (libsql_vector_idx is a
libSQL/sqld fork feature; DiskANN was closed not-planned upstream in
#832). The test asserts cross-book isolation and nearest-first ordering
using only `WHERE book_hash = ?` and `ORDER BY` — no DDL, no identifier
interpolation, no index plumbing.
Also adds bench/ harness for manual perf checks:
pnpm bench [name] run benchmarks (refuses in CI)
pnpm bench --list list available benchmarks
pnpm bench --no-record skip results.jsonl append
pnpm bench --force override the CI guard
Uses Node 24's --experimental-strip-types so no tsx devDep is needed.
Appends one JSON line per run to bench/results.jsonl (gitignored, local
history; share by pasting tabular stdout into PRs/issues). Explicitly
NOT in CI — shared-tenant variance makes synthetic-benchmark regression
detection unreliable; production telemetry (reedy_metrics, plan §M1.9)
is the right tool for that.
First benchmark: vector-retrieval. Measured on M1 Pro:
400 chunks × 384 dim → 0.35 ms / query
400 chunks × 768 dim → 0.45 ms / query
2000 chunks × 768 dim → 2.23 ms / query
10000 chunks × 768 dim → 14.00 ms / query
400 chunks × 1536 dim → 0.70 ms / query
Per-chunk cost ~1.1 µs at 768 dim = ~1.4 ns/dim. NEON-class on
Apple Silicon, ~50× faster than scalar — confirms SIMD acceleration is
active in 0.6.0-pre.28. Per-query latency stays sub-ms at Reedy MVP
corpus sizes; the ceiling is ~10K chunks per book before phone-class
hardware notices.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Set up WebDriver-based testing for the Tauri app with two tiers:
- Vitest browser-mode tests (*.tauri.test.ts) running inside the Tauri WebView
for plugin IPC testing (libsql, smoke tests)
- WDIO E2E tests (*.e2e.ts) for UI-level interaction testing
Key changes:
- Add webdriver Cargo feature gating tauri-plugin-webdriver
- Add runtime capability for remote URLs (webdriver builds only)
- Add vitest.tauri.config.mts and wdio.conf.ts connecting to embedded
WebDriver server on port 4445
- Add shared tauri-invoke helper for IPC from Vitest iframe context
- Add testing documentation in docs/testing.md
Huang Xin