Files
readest/apps/readest.koplugin/library/cloud_covers.lua
T
Huang Xin ed8956e9ed feat(koplugin): Readest Library view in KOReader (#4056)
* feat(koplugin/library): data layer + busted harness + design doc

- LibraryStore: per-user SQLite index merging cloud + local books by
  partial-md5 hash. listBooks/listBookshelfBooks/listBookshelfGroups,
  upsertBook with cloud_present/local_present OR-merge + _force/clear
  sentinels, parseSyncRow, getChangedBooks for tombstone push.
- EXTS table mirroring web's document.ts.
- busted harness with KOReader stubs (G_reader_settings, DataStorage,
  lua-ljsqlite3 against :memory:); spec_helper, exts_spec, smoke_spec,
  librarystore_spec covering schema, sort, group nesting, dedupe.
- Library design doc + spec README.
- pnpm test:lua wired through root + app package.json; lint-koplugin
  recurses into library/ + spec/.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(koplugin/library): cloud sync — push, pull, upload, delete, downloads

- Spore methods: pullBooks (incremental /sync), getDownloadUrl,
  getUploadUrl, listFiles, deleteFile.
- syncbooks.lua: pushBook + pushChangedBooks (advances watermark to
  max(updated_at, deleted_at)), syncBooks(opts, mode) for push/pull/both,
  downloadBook + downloadCover (sync socket.http with file sink; cover
  download via fork+poll with single-slot queue + visible-page filter +
  coalesced refresh), uploadBook (presigned-PUT flow + best-effort
  cover), deleteCloudFiles (list-then-delete-each, mirrors
  cloudService.deleteBook).
- SyncAuth.withFreshToken wrapper resolves the ensureClient race; 401/403
  unified across syncconfig + syncannotations.
- Cloud + local book covers shared by partial-md5 hash; cover.png cached
  at <settings>/readest_covers/<hash>.png with sentinel for known 404s.
- syncbooks_spec covers row-to-wire conversion + file_key shape.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(koplugin/library): local discovery — sidecar scanner + cover provider

- localscanner.lightScan iterates ReadHistory entries, reads
  partial_md5_checksum from .sdr sidecars, and upserts local rows.
  Slow filesystem walks deferred to fullSidecarWalk (24h-gated).
- coverprovider wraps BookInfoManager:getBookInfo for local books with
  graceful FakeCover fallback when coverbrowser is absent.
- localscanner_spec + coverprovider_spec.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(koplugin/library): UI — widget, item, view menu, FileManager hooks

- librarywidget: full-screen Menu mixed in with CoverMenu + Mosaic/List
  per zen_ui's group_view pattern. Title-bar tap → view menu, search via
  left icon, drill-in/back for grouping. Async cloud sync deferred via
  scheduleIn so the menu paints before HTTP fires.
- libraryitem: BIM patch for cloud-only (readest-cloud://) and group
  (readest-group://) URIs; group-cover composer (2x2 mosaic for grid,
  same in list) with cache key derived from the actual first-N hashes
  for content-based invalidation; ListMenuItem update + paintTo patches
  for wider list-mode cover strip and cloud-up/down icon overlay.
- libraryviewmenu: ButtonDialog with View/Group by/Sort by/Actions.
  Default Group by = Groups (parity with web), values authors/groups.
- librarypaint: partial-page e-ink repaint shim adapted from zen_ui.
- main.lua: Library menu entry, dispatcher actions (Open Library / Push
  / Pull as general; progress + annotations as reader-only),
  "Add to Readest" button in FileManager's long-press file dialog
  (dedupe by partial_md5; bumps updated_at when present, inserts a
  fresh local-only row otherwise; un-tombstones via _clear_fields).
  Push books on Library open when auto_sync is on, pull-only otherwise.
- Long-press action sheet with Readest BookDetailView parity:
  Remove from Cloud & Device / Cloud Only / Device Only,
  Upload to Cloud, Download Book / Cover / All.
- Cloud-down + cloud-up SVG icons (LiaCloudDownloadAltSolid /
  LiaCloudUploadAltSolid) painted in the right-side wpageinfo slot.
- i18n catalog updated for new strings.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor(koplugin/library): split libraryitem into focused modules

libraryitem.lua had grown to 1018 lines mixing five unrelated
concerns. Split along the natural seams:

  cloud_covers — readest-cloud:// URI scheme, on-disk <hash>.png
                 cache, single-slot async download queue, visible-page
                 filter
  group_covers — readest-group:// URI scheme, 2x2 mosaic composer with
                 content-derived cache key (first-N hashes), cell
                 layout table
  cloud_icons  — bundled cloud-up/cloud-down SVG loader, IconWidget
                 cache, paint-overlay positioning
  list_strip   — list-mode group row builder (4-cover wider strip
                 replacing ListMenu's square cover slot)
  bim_patch    — BookInfoManager:getBookInfo router (cloud / group /
                 local) + ListMenuItem update + paintTo patches; owns
                 the _library_local_paths set and orig BIM reference

libraryitem.lua is now 141 lines: just the entry-table constructors
(entry_from_row, entry_from_group, entry_back) plus thin install /
set_visible_hashes delegates. Each new module is 88-216 lines.

No behavior change — same 113 specs pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(koplugin): co-locate dev tooling, ship-zip exclusions, CI job split

- apps/readest.koplugin/scripts/build-koplugin.mjs: local sideloading
  build with the same exclusions the release workflow uses.
- Move lint-koplugin + test-koplugin from apps/readest-app/scripts/ to
  apps/readest.koplugin/scripts/. All koplugin dev tooling now lives
  with the koplugin and is excluded from the published release zip.
- Rename to .mjs so Node treats them as ESM without the reparse warning
  (the i18n CommonJS scripts stay .js).
- Release workflow: zip -r exclusions for scripts/, docs/, spec/,
  .busted so dev artifacts don't ship to end users.
- PR workflow: split build_web_app into build_web_app + test_web_app
  for parallelism. The test job installs luarocks + busted +
  lsqlite3complete and runs pnpm test:lua. test-koplugin.mjs now
  hard-fails (instead of soft-skipping) when CI=true and a tool is
  missing — a broken CI toolchain previously exited 0 silently.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 23:12:34 +02:00

214 lines
8.1 KiB
Lua

-- cloud_covers.lua
-- Per-book cover lifecycle for cloud-only rows. Owns the on-disk
-- <hash>.png cache, the synthetic readest-cloud:// URI scheme, and
-- the single-slot async download queue that fetches missing covers
-- from Readest storage when their cells become visible.
--
-- The disk cache is shared with hybrid (cloud+local) rows so a
-- previously-downloaded cloud cover gets reused for the local
-- presentation of the same hash without a re-extraction pass.
local logger = require("logger")
local M = {}
M.URI_PREFIX = "readest-cloud://"
-- Synthetic-info metadata cache: keyed by either hash (cloud-only
-- entries) or the full URI string (group entries). Values:
-- { title, author }
-- The patched BIM reads these to fill in title/authors so MosaicMenu's
-- FakeCover renders meaningful text instead of "?".
local _meta = {}
-- Download lifecycle state
local _cover_pending = {} -- hash → true while a download is in flight
local _missing_covers = {} -- hash → true after a 404 (don't keep retrying)
local _visible_hashes = nil -- set of hashes on the current Menu page; nil = no filter
local _refresh_pending = false -- coalesces multiple cover-completion refreshes
local _download_queue = {} -- FIFO list of pending hashes
local _downloading = false -- gate: only one socket.http active at a time
-- Sync-auth opts (set via M.set_opts at install time). Holds
-- sync_auth, sync_path, settings — needed to drive syncbooks.downloadCover.
local _opts = nil
function M.set_opts(opts)
_opts = opts
end
function M.set_meta(key, meta)
_meta[key] = meta
end
function M.get_meta(key)
return _meta[key] or {}
end
function M.covers_dir()
local DataStorage = require("datastorage")
return DataStorage:getSettingsDir() .. "/readest_covers"
end
local function cover_path_for(hash)
return M.covers_dir() .. "/" .. hash .. ".png"
end
-- Strip the .ext suffix so callers get just the partial-md5. The .ext
-- is encoded into the URI so listmenu's filemanagerutil.splitFileNameType
-- returns a non-nil filetype (listmenu.lua:316); without it the
-- right-column composition crashes on string concat.
function M.hash_from_uri(filepath)
local rest = filepath:sub(#M.URI_PREFIX + 1)
return (rest:match("^([^.]+)") or rest)
end
-- Load <hash>.png from disk into a fresh blitbuffer. Returns nil if the
-- file doesn't exist or fails to decode. Caller owns the bb (ImageWidget
-- will free it). No in-memory cache — ImageWidget treats the bb as
-- disposable, so sharing one across paints leads to use-after-free.
function M.load_cover_bb(hash)
local lfs = require("libs/libkoreader-lfs")
local path = cover_path_for(hash)
if lfs.attributes(path, "mode") ~= "file" then return nil end
local ok, RenderImage = pcall(require, "ui/renderimage")
if not ok then return nil end
local ok2, bb = pcall(RenderImage.renderImageFile, RenderImage, path, false)
if not ok2 or not bb then return nil end
return bb
end
-- "<hash8> '<title>'" formatted log tag — searchable by either id.
local function tag_for(hash)
local meta = _meta[hash] or {}
return hash:sub(1, 8) .. " '" .. tostring(meta.title or "?") .. "'"
end
-- Pump the next entry off _download_queue. Re-entrant-safe via the
-- _downloading gate. Filters known-404 hashes and hashes that have
-- scrolled off-screen since they were enqueued.
local function process_queue()
if _downloading then return end
local hash
repeat
hash = table.remove(_download_queue, 1)
if not hash then return end
if _missing_covers[hash] then
_cover_pending[hash] = nil
hash = nil
elseif _visible_hashes and not _visible_hashes[hash] then
logger.dbg("ReadestLibrary cover dequeue skip: " .. tag_for(hash)
.. " no longer on visible page")
_cover_pending[hash] = nil
hash = nil
end
until hash
_downloading = true
logger.info("ReadestLibrary cover download: starting " .. tag_for(hash))
local syncbooks = require("library.syncbooks")
syncbooks.downloadCover(
{ hash = hash },
{
sync_auth = _opts and _opts.sync_auth,
sync_path = _opts and _opts.sync_path,
settings = _opts and _opts.settings,
covers_dir = M.covers_dir(),
},
function(success, path_or_err, status)
_cover_pending[hash] = nil
_downloading = false
if not success then
if status == 404 then
_missing_covers[hash] = true
logger.info("ReadestLibrary cover " .. tag_for(hash)
.. " — no cover on server (404), won't retry")
else
logger.warn("ReadestLibrary cover " .. tag_for(hash)
.. " download failed: " .. tostring(path_or_err)
.. " status=" .. tostring(status))
end
else
logger.info("ReadestLibrary cover " .. tag_for(hash)
.. " saved → " .. tostring(path_or_err))
-- Coalesce refresh: multiple covers landing in the same
-- tick still get one repaint, not N flickering redraws.
if not _refresh_pending then
_refresh_pending = true
local UIManager = require("ui/uimanager")
UIManager:nextTick(function()
_refresh_pending = false
local ok, LibraryWidget = pcall(require, "library.librarywidget")
if ok and LibraryWidget._menu then LibraryWidget.refresh() end
end)
end
end
-- Yield to the UI loop before pumping the next one.
local UIManager = require("ui/uimanager")
UIManager:nextTick(process_queue)
end)
end
-- Idempotent against in-flight requests and known-404 hashes. Only
-- fires for hashes the caller has marked as visible via M.set_visible_hashes
-- — otherwise paint stragglers / poll loops for off-screen items would
-- queue downloads the user can't see.
function M.trigger_download(hash)
if _cover_pending[hash] then
logger.dbg("ReadestLibrary cover skip: " .. tag_for(hash) .. " already in flight")
return
end
if _missing_covers[hash] then
logger.dbg("ReadestLibrary cover skip: " .. tag_for(hash) .. " known 404")
return
end
if not _opts or not _opts.sync_auth then
logger.warn("ReadestLibrary cover skip: " .. tag_for(hash)
.. " — set_opts not called yet")
return
end
if _visible_hashes and not _visible_hashes[hash] then
logger.dbg("ReadestLibrary cover skip: " .. tag_for(hash) .. " not on visible page")
return
end
_cover_pending[hash] = true
table.insert(_download_queue, hash)
logger.dbg("ReadestLibrary cover queued: " .. tag_for(hash)
.. " (queue len=" .. #_download_queue .. ")")
process_queue()
end
-- Set of hashes on the current Menu page; only these may trigger
-- downloads. Pass nil to disable the filter (e.g. when the Library
-- closes and the patched BIM might still be invoked from elsewhere).
function M.set_visible_hashes(menu, cloud_only_flag)
if not menu then
logger.dbg("ReadestLibrary set_visible_hashes: cleared (menu nil)")
_visible_hashes = nil
return
end
local set = {}
local count = 0
local page = menu.page or 1
local perpage = menu.perpage or 1
local items = menu.item_table or {}
local first = (page - 1) * perpage + 1
local last = math.min(first + perpage - 1, #items)
for i = first, last do
local entry = items[i]
if entry and entry[cloud_only_flag] and type(entry.file) == "string" then
local hash = M.hash_from_uri(entry.file)
set[hash] = true
count = count + 1
end
end
_visible_hashes = set
logger.info("ReadestLibrary set_visible_hashes: page=" .. page
.. " range=" .. first .. ".." .. last
.. " cloud_only_visible=" .. count
.. " (item_table size=" .. #items .. ")")
end
return M