Skip to main content

​
Prerequisites

  • Python 3.9 or higher
  • Git
  • uv — the package manager used by this project

​
Setup

1

Clone the repository

git clone https://github.com/lncrawl/lightnovel-crawler
cd lightnovel-crawler
2

Install uv

If you don’t have uv installed, the Makefile will install it for you during the next step. To install it manually:
# macOS and Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

# Windows (PowerShell)
powershell -NoProfile -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
See the uv installation docs for more options.
3

Install dependencies

Run the install target, which syncs submodules and installs all Python dependencies including dev tools:
make install
This runs uv sync --extra dev under the hood. You can also run that command directly if you prefer.
4

Run the server

Start the backend server:
make start
# or
uv run python -m lncrawl
The server starts at http://localhost:8080 by default.
5

Enable auto-reload (optional)

For active development, run in watch mode. The server restarts automatically when you change a Python file:
make watch

​
Makefile command reference

All common tasks are available as Makefile targets. Run them from the repository root.
CommandDescription
make setupSync submodules and install uv if not present
make installInstall all Python dependencies (uv sync --extra dev)
make upgradeUpgrade all dependencies to their latest allowed versions
make startStart the backend server
make watchStart the backend server with auto-reload on file changes
make lintRun black, isort, and flake8 checks
make buildFull build: version check, install, wheel, and exe
make build-wheelBuild a Python wheel (.whl) only
make build-exeBuild a standalone executable via PyInstaller
make add-dep PKGAdd a runtime dependency (e.g. make add-dep httpx)
make add-dev PKGAdd a dev-only dependency (e.g. make add-dev black)
make rm-dep PKGRemove a runtime dependency
make rm-dev PKGRemove a dev-only dependency
make docker-buildBuild the Docker application image
make docker-upStart the stack with Docker Compose (background)
make docker-downStop the Docker Compose stack
make docker-logsStream logs from running containers
make cleanRemove .venv, build, dist, and cache directories
make pullGit pull with rebase and update submodules
make versionPrint the current version from lncrawl/VERSION

​
Linting

Run the full lint suite before opening a pull request: 
make lint
This runs three checks in sequence:
  1. black — code formatting
  2. isort — import ordering
  3. flake8 — style and error checking (configured in .flake8)
If black or isort report formatting issues, run them directly to auto-fix: 
uv run black .
uv run isort .

​
Building

# Runs version check, install, wheel, and exe
make build

​
Docker development

You can also develop and test using Docker: 
# Build the base image (Calibre + dependencies) then the app image
make docker-build

# Start the stack in the background
make docker-up

# Stop the stack
make docker-down

# Stream container logs
make docker-logs
Docker Compose configuration is at scripts/local-compose.yml.

​
Managing dependencies

Add and remove dependencies with uv through the Makefile targets. This keeps pyproject.toml and uv.lock in sync automatically. 
# Add a runtime dependency
make add-dep httpx

# Add a dev-only dependency
make add-dev black

# Remove a runtime dependency
make rm-dep httpx

# Remove a dev-only dependency
make rm-dev black

​
Project structure

Key directories and what they contain:
PathDescription
lncrawl/Main application package
lncrawl/core/Core base classes: Crawler, Scraper, TaskManager
lncrawl/templates/Crawler templates: soup/, browser/, madara.py, etc.
lncrawl/models/Data models: Chapter, Volume, Novel, SearchResult
lncrawl/server/FastAPI server and REST API endpoints
lncrawl/services/Application services: binder, sources registry, jobs
lncrawl/dao/Database access objects (SQLAlchemy/SQLModel)
sources/All source crawlers, organized by language
sources/_examples/Template examples for new crawlers
sources/en/English-language crawlers, split by first letter
sources/multi/Multi-language crawlers
scripts/Build and Docker scripts
All core services (config, db, http, sources, binder, jobs) are accessed through the AppContext singleton (ctx). See lncrawl/context.py for details.

Build docs developers (and LLMs) love

Get started for freeTalk to us