* feat(sync): decouple the incremental-pull cursor from updated_at (#4678) `books.updated_at` was overloaded as both the incremental-pull cursor (`GET /api/sync?since=…` filters `updated_at > since`, devices keep one global `max(updated_at)` watermark) and the library "date read" sort key. A server-resolved reading-status merge had to be written with a timestamp greater than every peer's global cursor to propagate, which forced `updated_at = now()` and reordered the date-read library by sync-processing time (the #4677 symptom). Introduce a server-assigned `synced_at` column on `books`, stamped by a `BEFORE INSERT OR UPDATE` trigger on every write, used only as the pull cursor. `updated_at` stays pure client event time used only for sorting. - Migration 016 + baseline schema: add `synced_at` (NOT NULL DEFAULT now()), index `(user_id, synced_at)`, trigger `set_books_synced_at`. Backfill `synced_at = updated_at` before creating the trigger so existing devices' cursors hand over without a re-sync storm. - GET: books filters/orders on `synced_at > since` (a delete bumps synced_at, so the deleted_at clause is dropped); configs/notes stay on updated_at. - POST: extract `buildStatusPropagationRow` and drop the `updated_at = now()` bump — the trigger advances synced_at so peers re-pull the status change while updated_at (the sort key) stays put. - Client `computeMaxTimestamp` keys on synced_at, falling back to updated_at/deleted_at for pre-migration servers and configs/notes. Backward-compatible: `synced_at >= updated_at` always, so `synced_at > since` is a strict superset of `updated_at > since` — old web clients and the koplugin keep working with no data loss (at worst a redundant idempotent re-pull of rare server-merged rows). The koplugin's shared pull/push cursor is left untouched; a proper split is a follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sync): make the books synced_at backfill safe for large live tables (#4678) The single `UPDATE … WHERE synced_at IS NULL` deadlocked on a 3.8M-row production `books` table: it rewrites every row in one transaction while the live /api/sync push path upserts books rows, and the two lock rows in opposite orders. `ALTER COLUMN … SET NOT NULL` (full-table ACCESS EXCLUSIVE scan) and a plain CREATE INDEX (write-blocking SHARE lock) compounded it. Rework migration 016 as an online migration (run via psql, not in a wrapping transaction): - backfill in small autocommitted batches via a procedure, using FOR UPDATE SKIP LOCKED so it never waits on an app-locked row; - CREATE INDEX CONCURRENTLY instead of a blocking build; - install the trigger last (so it can't clobber the updated_at backfill); - drop the hard SET NOT NULL (the default + trigger + backfill keep the column populated and the client falls back to updated_at); a NOT VALID CHECK + VALIDATE alternative is included, commented, for operators who want it. The baseline schema.sql (fresh, empty installs) keeps the simple inline NOT NULL DEFAULT now() + trigger. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Self-Hosting with Docker/Podman with Compose
Stack
| service | Image | Description |
|---|---|---|
| client | ghcr.io/readest/readest |
readest frontend |
| db | supabase/postgres |
psql db with supabase extensions |
| kong | kong:2.8.1 |
api gateway routing requests to supabase services |
| auth | supabase/gotrue:v2.185.0 |
auth service (email, JWT) |
| rest | postgrest/postgrest:v14.3 |
psql rest api |
| minio | minio/minio |
s3 storage |
| minio-setup | minio/mc |
helper container to create s3 buckets |
Exposed ports
| Port | Service |
|---|---|
3000 |
readest |
8000 |
kong API gateway |
9000 |
MinIO S3 API |
9001 |
MinIO console UI |
Running with Docker/Podman Compose
1. setup .env
cp docker/.env.example docker/.env
update docker/.env:
- update
POSTGRES_PASSWORDto a strong password (32+ chars) - update
JWT_SECRETto a random secret (32+ chars) - regenerate
ANON_KEYandSERVICE_ROLE_KEYas HS256 JWTs signed with yourJWT_SECRET(use jwt.io or a similar tool):ANON_KEYpayload:{"role": "anon"}SERVICE_ROLE_KEYpayload:{"role": "service_role"}
- set
MINIO_ROOT_PASSWORDto a strong password
2. Start the Stack (pull prebuilt client image)
run from the docker/ directory:
cd docker
docker compose up -d
this pulls ${READEST_IMAGE} (default: ghcr.io/readest/readest:latest) instead of building the client locally.
the web client now reads SUPABASE_PUBLIC_URL, SUPABASE_ANON_KEY, API_BASE_URL, OBJECT_STORAGE_TYPE, STORAGE_FIXED_QUOTA, and TRANSLATION_FIXED_QUOTA from runtime
container env, so custom self-hosted values work with pulled images.
if you prefer Docker Hub, set READEST_IMAGE in docker/.env, for example:
READEST_IMAGE=docker.io/your-dockerhub-username/readest:latest
replace your-dockerhub-username with the Docker Hub namespace that publishes your readest image.
for official images, use the namespace configured for this repository's Docker Hub publishing secrets.
published tags:
latest: rolling image from the default branch and from release events<release-tag>(for examplev1.2.3): published from release eventsmain: rolling image from the default branchsha-<commit>: immutable commit tag
Build locally instead of pulling
Prerequisites for local builds: the
packages/foliate-jsandpackages/simplecc-wasmgit submodules must be initialized before building:git submodule update --init packages/foliate-js packages/simplecc-wasmIn GitHub Codespaces this is done automatically via
.devcontainer/devcontainer.json.
cd docker
docker compose -f compose.yaml -f compose.build.yaml up --build -d
3. Access
- Readest app:
http://localhost:3000 - MinIO console:
http://localhost:9001(login withMINIO_ROOT_USER/MINIO_ROOT_PASSWORD)
Hot Reload (development)
Prerequisites: submodules must be initialized (see above).
to develop using the compose stack, use compose.dev.yaml which sets the build target to development-stage (Next.js dev server) and mounts your local repo for hot reload:
cd docker
docker compose -f compose.yaml -f compose.dev.yaml up --build -d
the first mount overlays your local repo into the container. the remaining anonymous volumes shadow the directories that were pre-built inside the image, so the container's installed deps and vendor assets are used instead of what's on your host.
Stop the Stack
cd docker
docker compose down
to also remove volumes (database and storage data):
cd docker
docker compose down -v
Building the Dockerfile standalone
docker build \
--target production-stage \
--build-arg NEXT_PUBLIC_APP_PLATFORM=web \
-t readest-client \
.
run the built image:
docker run -p 3000:3000 \
-e SUPABASE_URL=http://host.docker.internal:8000 \
-e SUPABASE_PUBLIC_URL=http://localhost:8000 \
-e SUPABASE_ANON_KEY=<anon-key> \
-e SUPABASE_ADMIN_KEY=<service-role-key> \
-e API_BASE_URL=http://localhost:3000 \
-e OBJECT_STORAGE_TYPE=s3 \
-e S3_ENDPOINT=http://host.docker.internal:9000 \
-e S3_PUBLIC_ENDPOINT=http://localhost:9000 \
-e S3_REGION=us-east-1 \
-e S3_BUCKET_NAME=readest-files \
-e S3_ACCESS_KEY_ID=<minio-user> \
-e S3_SECRET_ACCESS_KEY=<minio-password> \
-e STORAGE_FIXED_QUOTA=1073741824 \
-e TRANSLATION_FIXED_QUOTA=50000 \
readest-client
on Linux, some Docker setups do not resolve host.docker.internal by default.
in that case, either replace it with your host IP or run with:
--add-host=host.docker.internal:host-gateway.