Files
readest/apps/readest-app/src/services/ingestService.ts
T
Huang Xin 0b18de0581 feat(send): Send to Readest — multi-channel capture into your library (#4230)
* feat(send): Send to Readest — multi-channel capture into your library

A Send-to-Kindle equivalent: email, web-upload, share, or one-click
capture books and articles into the cloud library; they sync to every
device.

Architecture (client-side processing): out-of-app channels drop a raw
payload into a per-user send_inbox; Readest clients drain it through one
shared ingestService.ingestFile(). The server never parses or converts.

- ingestService.ingestFile() — channel-agnostic import orchestration
  extracted from library/page.tsx (DI-based, forceUpload support).
- send_addresses / send_allowed_senders / send_inbox tables + RLS + 4
  SECURITY DEFINER claim/lease RPCs (migration 012_send_to_readest.sql).
- Conversion subsystem (DOCX/RTF/HTML/article/TXT -> EPUB) in a Web Worker.
- send-email Cloudflare Email Worker; inbox-drainer controller +
  useInboxDrainer hook; /api/send/* routes.
- Send to Readest settings panel: inbound address, approved-sender
  allowlist, recent activity, per-device drain toggle.
- /send web page (file drop + article URL) + SSRF-guarded fetch-url proxy.
- OS-shared files routed through ingestFile; Manifest V3 browser extension.

Security: inbox state changes only via SECURITY DEFINER RPCs (clients get
SELECT-only on send_inbox); approved-sender allowlist gates email;
SSRF guard on the one server-side URL fetch; inbox payload signed URLs
authorize against send_inbox.user_id.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: run format:check in the pre-push hook

Biome format checking is fast (~0.4s), so gate pushes on it too — catches
mis-formatted files that bypassed the staged-only pre-commit hook before
they reach CI.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(send): address CodeQL security findings

- ReDoS (senders.ts): the email regex had ambiguous quantifiers around
  the literal dot. Rewrote it linear-time (domain labels exclude '.')
  and cap the input at 254 chars.
- XSS (convertToEpub.ts): run untrusted HTML through DOMPurify
  (sanitizeForParsing — keeps document structure) before DOMParser, so
  title extraction and Readability never parse executable markup.
- SSRF (fetch-url.ts): harden the host guard — block bare single-label
  hostnames, IPv4-mapped IPv6, CGNAT/benchmark/multicast ranges, and the
  unspecified address. DNS rebinding stays a documented residual risk.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-19 20:06:52 +02:00

80 lines
2.5 KiB
TypeScript

import type { Book, BookLookupIndex } from '@/types/book';
import type { AppService } from '@/types/system';
import type { SystemSettings } from '@/types/settings';
import { transferManager } from '@/services/transferManager';
export interface IngestFileDeps {
appService: AppService;
settings: SystemSettings;
isLoggedIn: boolean;
}
export interface IngestFileOptions {
/** A file path (desktop/mobile) or a File object (web). */
file: File | string;
/** Current library, used by importBook for dedup. */
books: Book[];
/** Pre-built lookup index for O(1) dedup during batch imports. */
lookupIndex?: BookLookupIndex;
/** Collection to place the book in. */
groupId?: string;
groupName?: string;
/** Tag parsed from a Send-to-Readest email subject (`#scifi`). */
subjectTag?: string;
/** Upload to the cloud even when the user has disabled autoUpload. */
forceUpload?: boolean;
/** Transient import (not stored long-term) — never uploaded. */
transient?: boolean;
}
/**
* Channel-agnostic single-file ingestion. Every capture channel — local library
* import, the /send page, the inbox drainer — calls this so a sent book behaves
* exactly like a locally-imported one.
*
* Persistence (`updateBooks` / `saveLibraryBooks`) and the sync push stay with
* the caller on purpose: batch importers save once per batch, single-item
* callers save per item. The shared logic that must NOT diverge — importing,
* group/tag metadata, the upload decision — lives here.
*/
export async function ingestFile(
opts: IngestFileOptions,
deps: IngestFileDeps,
): Promise<Book | null> {
const { appService, settings, isLoggedIn } = deps;
const book = await appService.importBook(opts.file, opts.books, {
lookupIndex: opts.lookupIndex,
transient: opts.transient,
});
if (!book) return null;
if (opts.groupId) {
book.groupId = opts.groupId;
book.groupName = opts.groupName;
}
const tag = opts.subjectTag?.trim();
if (tag) {
const tags = book.tags ?? [];
if (!tags.includes(tag)) {
book.tags = [...tags, tag];
book.updatedAt = Date.now();
}
}
// Sent books force the upload so they reach the user's other devices even
// when autoUpload is off; normal library imports honor the setting.
// Transient imports are never uploaded.
if (
!opts.transient &&
isLoggedIn &&
!book.uploadedAt &&
(opts.forceUpload || settings.autoUpload)
) {
transferManager.queueUpload(book);
}
return book;
}