Configuration

Lightnovel Crawler stores its settings in a JSON file. The file is created automatically on first run and updated whenever you change a setting.

Config file location

The config file is located at:

<APP_DIR>/config.json

The default APP_DIR is platform-specific, resolved via typer.get_app_dir("LNCrawl"):

Platform	Default path
Linux	`~/.config/LNCrawl/`
macOS	`~/Library/Application Support/LNCrawl/`
Windows	`%APPDATA%\LNCrawl\`

Set the LNCRAWL_DATA_PATH environment variable to override this directory. See Environment variables for details.

To use a custom config file path, pass --config <path> when running any lncrawl command, or set the LNCRAWL_CONFIG environment variable.

Config file format

The config file is a JSON object with top-level keys for each section:

{
  "app": {
    "openai_api_key": ""
  },
  "database": {
    "url": "sqlite:////home/user/.config/LNCrawl/sqlite.db",
    "pool_size": 10,
    "pool_timeout": 30,
    "pool_recycle": 3600,
    "connect_timeout": 10,
    "admin_email": "admin",
    "admin_password": "admin"
  },
  "crawler": {
    "can_use_browser": true,
    "selenium_grid": "",
    "ignore_images": false,
    "runner_concurrency": 5,
    "runner_cooldown": 1,
    "cleaner_cooldown": 1800,
    "runner_reset_interval": 14400,
    "disk_size_limit_mb": 0
  },
  "server": {
    "base_url": "http://localhost:8080",
    "token_secret": "<auto-generated UUID>",
    "token_algorithm": "HS256",
    "token_expiry_minutes": 10080
  },
  "mail": {
    "smtp_server": "localhost",
    "smtp_port": 1025,
    "smtp_username": "",
    "smtp_password": "",
    "smtp_sender": ""
  }
}

Do not edit the config file while Lightnovel Crawler is running. The application writes to it atomically but reads it on startup.

Using `lncrawl config`

The lncrawl config command lets you view and modify settings without editing the file directly.

View all settings

lncrawl config view

Add --format json or --format yaml to change the output format:

lncrawl config view --format json

Get a specific value

lncrawl config get <section> <key>

Examples:

lncrawl config get app openai_api_key
lncrawl config get crawler runner_concurrency
lncrawl config get mail smtp_port

Omit the key to select it interactively from a menu.

Set a value

lncrawl config set <section> <key> <value>

Examples:

lncrawl config set server base_url "https://example.com"
lncrawl config set crawler runner_concurrency "10"
lncrawl config set mail smtp_port "587"

Omit the key or value to be prompted interactively.

Config sections

app — Application settings

General application settings.

Key	Type	Default	Description
`openai_api_key`	string	`""`	API key for OpenAI integrations.

lncrawl config set app openai_api_key "sk-..."

database — Database connection

Controls the database used to store novels, chapters, and jobs.

Key	Type	Default	Description
`url`	string	`sqlite:///<APP_DIR>/sqlite.db`	SQLAlchemy database URL. Supports SQLite, PostgreSQL, and MySQL.
`pool_size`	integer	`10`	Number of connections to maintain in the connection pool.
`pool_timeout`	integer	`30`	Seconds to wait before giving up on obtaining a connection.
`pool_recycle`	integer	`3600`	Recycle connections after this many seconds to prevent stale connections.
`connect_timeout`	integer	`10`	Timeout in seconds for establishing a database connection.
`admin_email`	string	`"admin"`	Email address for the default admin account.
`admin_password`	string	`"admin"`	Password for the default admin account.

Change admin_email and admin_password from their defaults before exposing the web server to a network.

Example database URLs:

sqlite:////home/user/sqlite.db
postgresql+psycopg://pguser:pgpass@postgres:5432/lncrawl
mysql+pymysql://user:password@mysql:3306/lncrawl

The DATABASE_URL environment variable overrides the url config key. See Environment variables.

crawler — Crawling behavior

Controls how the crawler fetches and processes content.

Key	Type	Default	Description
`can_use_browser`	boolean	`true`	Allow the crawler to use a headless browser (Chrome/Selenium) for JavaScript-heavy sites.
`selenium_grid`	string	`""`	URL of a remote Selenium Grid endpoint. Leave empty to use a local browser.
`ignore_images`	boolean	`false`	Skip downloading images when crawling chapters.
`runner_concurrency`	integer	`5`	Number of crawler jobs that can run concurrently.
`runner_cooldown`	integer	`1`	Seconds to wait between starting consecutive crawler jobs.
`cleaner_cooldown`	integer	`1800`	Seconds between scrubber/cleanup job runs (default: 30 minutes).
`runner_reset_interval`	integer	`14400`	Seconds between scheduler restarts (default: 4 hours).
`disk_size_limit_mb`	integer	`0`	Maximum disk space in MB for crawled data. `0` means unlimited.

# Increase concurrency for faster crawling
lncrawl config set crawler runner_concurrency "10"

# Disable browser for environments without Chrome
lncrawl config set crawler can_use_browser "false"

# Point to a remote Selenium Grid
lncrawl config set crawler selenium_grid "http://selenium-hub:4444"

server — Web server settings

Controls the built-in HTTP server and authentication tokens.

Key	Type	Default	Description
`base_url`	string	`"http://localhost:8080"`	Public base URL of the server. Used for generating absolute links.
`token_secret`	string	auto-generated UUID	Secret key used to sign authentication tokens. Regenerated if not set.
`token_algorithm`	string	`"HS256"`	JWT signing algorithm.
`token_expiry_minutes`	integer	`10080`	Token validity in minutes (default: 7 days).

Set token_secret to a long random string in production. The auto-generated value changes if the config file is reset.

lncrawl config set server base_url "https://myserver.example.com"
lncrawl config set server token_expiry_minutes "1440"

mail — SMTP email settings

Controls outbound email delivery.

Key	Type	Default	Description
`smtp_server`	string	`"localhost"`	SMTP server hostname.
`smtp_port`	integer	`1025`	SMTP server port.
`smtp_username`	string	`""`	SMTP authentication username.
`smtp_password`	string	`""`	SMTP authentication password.
`smtp_sender`	string	`""`	From address used in outgoing emails.

lncrawl config set mail smtp_server "smtp.gmail.com"
lncrawl config set mail smtp_port "587"
lncrawl config set mail smtp_username "[email protected]"
lncrawl config set mail smtp_sender "[email protected]"

CLI flags override config

Some CLI flags set config values for the duration of a single command without persisting them to the file:

Flag	Effect
`--config <path>`	Load a specific config file instead of the default.
`--verbose` / `-l`	Increase log verbosity (`-l` = warn, `-ll` = info, `-lll` = debug).

Resetting config to defaults

To reset all settings to their defaults, delete the config file. Lightnovel Crawler recreates it with default values on the next run.

# Linux / macOS
rm ~/.config/LNCrawl/config.json

# Windows (PowerShell)
Remove-Item "$env:APPDATA\LNCrawl\config.json"

Deleting the config file also regenerates token_secret, which invalidates any existing authentication tokens.

Get Started

Usage

Sources

Contributing

Configuration

Config file location

Config file format

Using `lncrawl config`

Config sections

CLI flags override config

Resetting config to defaults

Build docs developers (and LLMs) love

Get Started

Usage

Configuration

Sources

Contributing

​Config file location

​Config file format

​Using lncrawl config

​Config sections

​CLI flags override config

​Resetting config to defaults

Build docs developers (and LLMs) love

Config file location

Config file format

Using `lncrawl config`

Config sections

CLI flags override config

Resetting config to defaults