forked from akai/readest
ed8956e9ed
* feat(koplugin/library): data layer + busted harness + design doc - LibraryStore: per-user SQLite index merging cloud + local books by partial-md5 hash. listBooks/listBookshelfBooks/listBookshelfGroups, upsertBook with cloud_present/local_present OR-merge + _force/clear sentinels, parseSyncRow, getChangedBooks for tombstone push. - EXTS table mirroring web's document.ts. - busted harness with KOReader stubs (G_reader_settings, DataStorage, lua-ljsqlite3 against :memory:); spec_helper, exts_spec, smoke_spec, librarystore_spec covering schema, sort, group nesting, dedupe. - Library design doc + spec README. - pnpm test:lua wired through root + app package.json; lint-koplugin recurses into library/ + spec/. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(koplugin/library): cloud sync — push, pull, upload, delete, downloads - Spore methods: pullBooks (incremental /sync), getDownloadUrl, getUploadUrl, listFiles, deleteFile. - syncbooks.lua: pushBook + pushChangedBooks (advances watermark to max(updated_at, deleted_at)), syncBooks(opts, mode) for push/pull/both, downloadBook + downloadCover (sync socket.http with file sink; cover download via fork+poll with single-slot queue + visible-page filter + coalesced refresh), uploadBook (presigned-PUT flow + best-effort cover), deleteCloudFiles (list-then-delete-each, mirrors cloudService.deleteBook). - SyncAuth.withFreshToken wrapper resolves the ensureClient race; 401/403 unified across syncconfig + syncannotations. - Cloud + local book covers shared by partial-md5 hash; cover.png cached at <settings>/readest_covers/<hash>.png with sentinel for known 404s. - syncbooks_spec covers row-to-wire conversion + file_key shape. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(koplugin/library): local discovery — sidecar scanner + cover provider - localscanner.lightScan iterates ReadHistory entries, reads partial_md5_checksum from .sdr sidecars, and upserts local rows. Slow filesystem walks deferred to fullSidecarWalk (24h-gated). - coverprovider wraps BookInfoManager:getBookInfo for local books with graceful FakeCover fallback when coverbrowser is absent. - localscanner_spec + coverprovider_spec. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(koplugin/library): UI — widget, item, view menu, FileManager hooks - librarywidget: full-screen Menu mixed in with CoverMenu + Mosaic/List per zen_ui's group_view pattern. Title-bar tap → view menu, search via left icon, drill-in/back for grouping. Async cloud sync deferred via scheduleIn so the menu paints before HTTP fires. - libraryitem: BIM patch for cloud-only (readest-cloud://) and group (readest-group://) URIs; group-cover composer (2x2 mosaic for grid, same in list) with cache key derived from the actual first-N hashes for content-based invalidation; ListMenuItem update + paintTo patches for wider list-mode cover strip and cloud-up/down icon overlay. - libraryviewmenu: ButtonDialog with View/Group by/Sort by/Actions. Default Group by = Groups (parity with web), values authors/groups. - librarypaint: partial-page e-ink repaint shim adapted from zen_ui. - main.lua: Library menu entry, dispatcher actions (Open Library / Push / Pull as general; progress + annotations as reader-only), "Add to Readest" button in FileManager's long-press file dialog (dedupe by partial_md5; bumps updated_at when present, inserts a fresh local-only row otherwise; un-tombstones via _clear_fields). Push books on Library open when auto_sync is on, pull-only otherwise. - Long-press action sheet with Readest BookDetailView parity: Remove from Cloud & Device / Cloud Only / Device Only, Upload to Cloud, Download Book / Cover / All. - Cloud-down + cloud-up SVG icons (LiaCloudDownloadAltSolid / LiaCloudUploadAltSolid) painted in the right-side wpageinfo slot. - i18n catalog updated for new strings. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(koplugin/library): split libraryitem into focused modules libraryitem.lua had grown to 1018 lines mixing five unrelated concerns. Split along the natural seams: cloud_covers — readest-cloud:// URI scheme, on-disk <hash>.png cache, single-slot async download queue, visible-page filter group_covers — readest-group:// URI scheme, 2x2 mosaic composer with content-derived cache key (first-N hashes), cell layout table cloud_icons — bundled cloud-up/cloud-down SVG loader, IconWidget cache, paint-overlay positioning list_strip — list-mode group row builder (4-cover wider strip replacing ListMenu's square cover slot) bim_patch — BookInfoManager:getBookInfo router (cloud / group / local) + ListMenuItem update + paintTo patches; owns the _library_local_paths set and orig BIM reference libraryitem.lua is now 141 lines: just the entry-table constructors (entry_from_row, entry_from_group, entry_back) plus thin install / set_visible_hashes delegates. Each new module is 88-216 lines. No behavior change — same 113 specs pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(koplugin): co-locate dev tooling, ship-zip exclusions, CI job split - apps/readest.koplugin/scripts/build-koplugin.mjs: local sideloading build with the same exclusions the release workflow uses. - Move lint-koplugin + test-koplugin from apps/readest-app/scripts/ to apps/readest.koplugin/scripts/. All koplugin dev tooling now lives with the koplugin and is excluded from the published release zip. - Rename to .mjs so Node treats them as ESM without the reparse warning (the i18n CommonJS scripts stay .js). - Release workflow: zip -r exclusions for scripts/, docs/, spec/, .busted so dev artifacts don't ship to end users. - PR workflow: split build_web_app into build_web_app + test_web_app for parallelism. The test job installs luarocks + busted + lsqlite3complete and runs pnpm test:lua. test-koplugin.mjs now hard-fails (instead of soft-skipping) when CI=true and a tool is missing — a broken CI toolchain previously exited 0 silently. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
150 lines
6.7 KiB
Lua
150 lines
6.7 KiB
Lua
-- localscanner_spec.lua
|
|
-- Unit-tests the pure helpers in library/localscanner.lua.
|
|
--
|
|
-- Out of scope: lightScan() and fullSidecarWalk() require live KOReader
|
|
-- modules (ReadHistory, DocSettings, FFIUtil.runInSubProcess); they're
|
|
-- exercised manually via the test matrix in docs/library-design.md.
|
|
|
|
require("spec_helper")
|
|
local lfs = require("lfs")
|
|
|
|
describe("library.localscanner", function()
|
|
local localscanner, datastorage
|
|
|
|
before_each(function()
|
|
require("spec_helper").reset()
|
|
package.loaded["library.localscanner"] = nil
|
|
localscanner = require("library.localscanner")
|
|
datastorage = require("datastorage")
|
|
end)
|
|
|
|
-- =====================================================================
|
|
-- sidecar_to_book_path: { "/foo/bar.sdr/metadata.epub.lua" → "/foo/bar.epub" }
|
|
-- =====================================================================
|
|
describe("sidecar_to_book_path", function()
|
|
it("strips .sdr/metadata.<ext>.lua and re-appends .ext to the parent", function()
|
|
assert.are.equal("/foo/bar.epub",
|
|
localscanner.sidecar_to_book_path("/foo/bar.sdr/metadata.epub.lua"))
|
|
assert.are.equal("/books/Asimov - Foundation.pdf",
|
|
localscanner.sidecar_to_book_path(
|
|
"/books/Asimov - Foundation.sdr/metadata.pdf.lua"))
|
|
assert.are.equal("/h/Dune.fb2",
|
|
localscanner.sidecar_to_book_path("/h/Dune.sdr/metadata.fb2.lua"))
|
|
end)
|
|
|
|
it("preserves multi-dot book names", function()
|
|
assert.are.equal("/x/Foo. Vol. 1.epub",
|
|
localscanner.sidecar_to_book_path("/x/Foo. Vol. 1.sdr/metadata.epub.lua"))
|
|
end)
|
|
|
|
it("returns nil for paths that aren't sidecar metadata", function()
|
|
assert.is_nil(localscanner.sidecar_to_book_path("/foo/bar.epub"))
|
|
assert.is_nil(localscanner.sidecar_to_book_path("/foo/bar.sdr/cover.png"))
|
|
assert.is_nil(localscanner.sidecar_to_book_path("/foo/random.lua"))
|
|
assert.is_nil(localscanner.sidecar_to_book_path(""))
|
|
assert.is_nil(localscanner.sidecar_to_book_path(nil))
|
|
end)
|
|
end)
|
|
|
|
-- =====================================================================
|
|
-- parse_sidecar: load a sidecar Lua file, extract partial_md5_checksum
|
|
-- + a few meta fields. Defensive against bad input.
|
|
-- =====================================================================
|
|
describe("parse_sidecar", function()
|
|
local function write(path, text)
|
|
local f = assert(io.open(path, "w"))
|
|
f:write(text)
|
|
f:close()
|
|
end
|
|
|
|
it("extracts partial_md5_checksum from a well-formed sidecar", function()
|
|
local sdir = datastorage:getSettingsDir() .. "/Foundation.sdr"
|
|
assert(lfs.mkdir(sdir))
|
|
local spath = sdir .. "/metadata.epub.lua"
|
|
write(spath, [[
|
|
return {
|
|
["partial_md5_checksum"] = "abc123def456",
|
|
["doc_props"] = {
|
|
["title"] = "Foundation",
|
|
["authors"] = "Isaac Asimov",
|
|
},
|
|
["page"] = 42,
|
|
}
|
|
]])
|
|
local out = localscanner.parse_sidecar(spath)
|
|
assert.are.equal("abc123def456", out.hash)
|
|
assert.are.equal("Foundation", out.title)
|
|
assert.are.equal("Isaac Asimov", out.author)
|
|
-- book_path computed from sidecar location
|
|
assert.are.equal(sdir:gsub("%.sdr$", ".epub"), out.file_path)
|
|
end)
|
|
|
|
it("returns nil for a sidecar without partial_md5_checksum", function()
|
|
local sdir = datastorage:getSettingsDir() .. "/Untracked.sdr"
|
|
assert(lfs.mkdir(sdir))
|
|
local spath = sdir .. "/metadata.epub.lua"
|
|
write(spath, "return { doc_props = { title = 'X' }, page = 1 }")
|
|
assert.is_nil(localscanner.parse_sidecar(spath))
|
|
end)
|
|
|
|
it("returns nil for a malformed Lua file", function()
|
|
local sdir = datastorage:getSettingsDir() .. "/Bad.sdr"
|
|
assert(lfs.mkdir(sdir))
|
|
local spath = sdir .. "/metadata.epub.lua"
|
|
write(spath, "return { -- unclosed table")
|
|
assert.is_nil(localscanner.parse_sidecar(spath))
|
|
end)
|
|
|
|
it("returns nil when the file does not exist", function()
|
|
assert.is_nil(localscanner.parse_sidecar(
|
|
datastorage:getSettingsDir() .. "/Missing.sdr/metadata.epub.lua"))
|
|
end)
|
|
|
|
it("survives a sidecar that returns a non-table", function()
|
|
local sdir = datastorage:getSettingsDir() .. "/Weird.sdr"
|
|
assert(lfs.mkdir(sdir))
|
|
local spath = sdir .. "/metadata.epub.lua"
|
|
write(spath, "return 'string instead of table'")
|
|
assert.is_nil(localscanner.parse_sidecar(spath))
|
|
end)
|
|
|
|
it("survives a sidecar that errors at load (e.g. forbidden global)", function()
|
|
local sdir = datastorage:getSettingsDir() .. "/Throws.sdr"
|
|
assert(lfs.mkdir(sdir))
|
|
local spath = sdir .. "/metadata.epub.lua"
|
|
write(spath, "error('boom'); return {}")
|
|
assert.is_nil(localscanner.parse_sidecar(spath))
|
|
end)
|
|
end)
|
|
|
|
-- =====================================================================
|
|
-- should_skip_dir: predicate for the recursive walk
|
|
-- =====================================================================
|
|
describe("should_skip_dir", function()
|
|
it("skips system / VCS / IDE dirs", function()
|
|
assert.is_true(localscanner.should_skip_dir(".git"))
|
|
assert.is_true(localscanner.should_skip_dir(".svn"))
|
|
assert.is_true(localscanner.should_skip_dir("node_modules"))
|
|
assert.is_true(localscanner.should_skip_dir(".Trash"))
|
|
assert.is_true(localscanner.should_skip_dir("$RECYCLE.BIN"))
|
|
assert.is_true(localscanner.should_skip_dir(".adobe-digital-editions"))
|
|
end)
|
|
|
|
it("skips macOS metadata dirs", function()
|
|
assert.is_true(localscanner.should_skip_dir(".Spotlight-V100"))
|
|
assert.is_true(localscanner.should_skip_dir(".fseventsd"))
|
|
end)
|
|
|
|
it("does NOT skip '.' / '..' (caller handles those)", function()
|
|
assert.is_false(localscanner.should_skip_dir("."))
|
|
assert.is_false(localscanner.should_skip_dir(".."))
|
|
end)
|
|
|
|
it("does NOT skip ordinary book directories", function()
|
|
assert.is_false(localscanner.should_skip_dir("Books"))
|
|
assert.is_false(localscanner.should_skip_dir("Sci-Fi"))
|
|
assert.is_false(localscanner.should_skip_dir("Foo.sdr")) -- the walk visits these
|
|
end)
|
|
end)
|
|
end)
|