Skip to main content

​
Overview

Jobs represent individual crawl operations for a domain. Each job crawls pages, collects performance data, and identifies issues like broken links and slow-loading pages.

​
Create Job

POST /v1/jobs
Authorization: Bearer <token>
Content-Type: application/json

{
  "domain": "example.com",
  "options": {
    "use_sitemap": true,
    "find_links": true,
    "max_pages": 100,
    "concurrency": 20
  }
}

​
Request Body

​
domain
string
required
The domain to crawl (e.g., “example.com”)
​
use_sitemap
boolean
default:true
Whether to discover pages from sitemap.xml
​
find_links
boolean
default:true
Whether to discover pages by crawling internal links
​
allow_cross_subdomain_links
boolean
default:true
Whether to follow links across subdomains
​
max_pages
integer
default:0
Maximum number of pages to crawl (0 = unlimited)
​
concurrency
integer
default:20
Number of concurrent requests (max 100)
​
source_type
string
Source that created the job (e.g., “dashboard”, “api”, “scheduler”)
​
source_detail
string
Additional detail about the source (e.g., “create_job”, “recurring”)

​
Response Fields

​
id
string
Unique job identifier
​
domain
string
The domain being crawled
​
domain_id
integer
Internal domain identifier
​
status
string
Job status: created, running, completed, failed, cancelled
​
organisation_id
string
Organisation that owns this job
​
created_at
string
ISO 8601 timestamp of job creation

​
List Jobs

GET /v1/jobs?page=1&limit=20&status=running
Authorization: Bearer <token>

​
Query Parameters

​
limit
integer
default:10
Number of results per page (max 100)
​
offset
integer
default:0
Number of results to skip
​
status
string
Filter by job status: created, running, completed, failed, cancelled
​
range
string
Date range filter (e.g., “7d”, “30d”)
​
tzOffset
integer
Timezone offset in minutes
​
include
string
Additional fields to include (comma-separated): stats, progress, domain

​
Response Fields

​
jobs
array
Array of job objects
​
pagination
object
Pagination metadata
​
pagination.limit
integer
Results per page
​
pagination.offset
integer
Current offset
​
pagination.total
integer
Total number of jobs
​
pagination.has_next
boolean
Whether there are more results

​
Get Job

GET /v1/jobs/{job_id}
Authorization: Bearer <token>

​
Path Parameters

​
job_id
string
required
Unique job identifier

​
Response Fields

​
progress
object
Job progress information
​
progress.total_tasks
integer
Total number of tasks
​
progress.completed_tasks
integer
Number of completed tasks
​
progress.failed_tasks
integer
Number of failed tasks
​
progress.skipped_tasks
integer
Number of skipped tasks
​
progress.percentage
number
Completion percentage
​
stats
object
Job statistics
​
stats.avg_response_time
integer
Average response time in milliseconds
​
stats.cache_hit_ratio
number
Cache hit ratio (0-1)
​
stats.total_bytes
integer
Total bytes transferred
​
started_at
string
ISO 8601 timestamp when job started (null if not started)
​
completed_at
string
ISO 8601 timestamp when job completed (null if not completed)

​
Cancel Job

POST /v1/jobs/{job_id}/cancel
Authorization: Bearer <token>

​
Path Parameters

​
job_id
string
required
Unique job identifier

​
Response Fields

​
status
string
Updated job status (will be “cancelled”)
​
cancelled_at
string
ISO 8601 timestamp when job was cancelled

​
Alternative: Delete Job

You can also cancel a job using DELETE: 
DELETE /v1/jobs/{job_id}
Authorization: Bearer <token>
This performs the same cancellation operation as POST /v1/jobs/{job_id}/cancel.

Build docs developers (and LLMs) love

Get started for freeTalk to us