Custom Webhooks

Overview

Adapt can send webhook notifications to your endpoints when crawl jobs complete, fail, or reach specific milestones. This enables you to integrate Adapt into your existing workflows, trigger downstream processes, or build custom notification systems. Key Features:

Event-driven notifications (job completed, failed, etc.)
HMAC signature validation for security
Automatic retry with exponential backoff
Idempotent payload design
Configurable timeout and rate limits

Custom webhook registration is coming soon. Currently, Adapt supports incoming webhooks from Webflow. Use this documentation to understand the webhook patterns for when custom webhooks become available.

Webhook Events

Adapt will send webhooks for the following events:

Job Completed

Triggered when a crawl job finishes successfully. Event: job.completed Payload:

{
  "event": "job.completed",
  "timestamp": "2024-03-03T14:45:00Z",
  "data": {
    "job_id": "job_123abc",
    "organisation_id": "org_456def",
    "domain": "example.com",
    "status": "completed",
    "total_tasks": 150,
    "completed_tasks": 150,
    "failed_tasks": 5,
    "stats": {
      "avg_response_time": 234,
      "cache_hit_ratio": 0.85,
      "total_bytes": 2048576
    },
    "started_at": "2024-03-03T14:30:00Z",
    "completed_at": "2024-03-03T14:45:00Z",
    "duration_seconds": 900
  },
  "signature": "sha256=a1b2c3d4e5f6..."
}

Job Failed

Triggered when a crawl job fails permanently. Event: job.failed Payload:

{
  "event": "job.failed",
  "timestamp": "2024-03-03T14:50:00Z",
  "data": {
    "job_id": "job_123abc",
    "organisation_id": "org_456def",
    "domain": "example.com",
    "status": "failed",
    "error": {
      "code": "timeout",
      "message": "Job exceeded maximum duration of 1 hour"
    },
    "total_tasks": 150,
    "completed_tasks": 45,
    "failed_tasks": 2,
    "started_at": "2024-03-03T13:50:00Z",
    "failed_at": "2024-03-03T14:50:00Z"
  },
  "signature": "sha256=a1b2c3d4e5f6..."
}

Job Progress

Triggered at 25%, 50%, 75% completion for long-running jobs. Event: job.progress Payload:

{
  "event": "job.progress",
  "timestamp": "2024-03-03T14:38:00Z",
  "data": {
    "job_id": "job_123abc",
    "organisation_id": "org_456def",
    "domain": "example.com",
    "status": "running",
    "progress": {
      "percentage": 50,
      "total_tasks": 150,
      "completed_tasks": 75,
      "failed_tasks": 2
    },
    "stats": {
      "avg_response_time": 245,
      "cache_hit_ratio": 0.83
    }
  },
  "signature": "sha256=a1b2c3d4e5f6..."
}

Security

HMAC Signature Validation

All webhook payloads include an HMAC-SHA256 signature in the signature field. Verify this signature to ensure the webhook came from Adapt. Signature Format:

sha256=<hex_encoded_hmac>

Validation Example (Go):

import (
    "crypto/hmac"
    "crypto/sha256"
    "encoding/hex"
    "encoding/json"
)

func validateWebhook(payload []byte, signature, secret string) bool {
    // Remove "sha256=" prefix
    signature = strings.TrimPrefix(signature, "sha256=")
    
    // Calculate expected signature
    h := hmac.New(sha256.New, []byte(secret))
    h.Write(payload)
    expectedSignature := hex.EncodeToString(h.Sum(nil))
    
    // Compare signatures (constant time)
    return hmac.Equal(
        []byte(expectedSignature),
        []byte(signature),
    )
}

Validation Example (Python):

import hmac
import hashlib

def validate_webhook(payload: bytes, signature: str, secret: str) -> bool:
    # Remove "sha256=" prefix
    signature = signature.replace("sha256=", "")
    
    # Calculate expected signature
    expected = hmac.new(
        secret.encode(),
        payload,
        hashlib.sha256
    ).hexdigest()
    
    # Compare signatures (constant time)
    return hmac.compare_digest(expected, signature)

Validation Example (Node.js):

const crypto = require('crypto');

function validateWebhook(payload, signature, secret) {
    // Remove "sha256=" prefix
    signature = signature.replace('sha256=', '');
    
    // Calculate expected signature
    const hmac = crypto.createHmac('sha256', secret);
    hmac.update(payload);
    const expected = hmac.digest('hex');
    
    // Compare signatures (constant time)
    return crypto.timingSafeEqual(
        Buffer.from(expected),
        Buffer.from(signature)
    );
}

Always use constant-time comparison functions to prevent timing attacks. Never use simple string equality (== or ===).

Test Signature Validation

From internal/api/webhook_test.go:16-103, Adapt includes comprehensive tests for signature validation:

func TestWebhookSignatureValidation(t *testing.T) {
    secret := "test-webhook-secret"
    
    tests := []struct {
        name           string
        payload        map[string]any
        signature      string
        expectedStatus int
    }{
        {
            name: "valid_signature",
            payload: map[string]any{
                "site":  "example.com",
                "event": "site_publish",
            },
            signature:      "", // Calculated in test
            expectedStatus: http.StatusOK,
        },
        {
            name: "invalid_signature",
            payload: map[string]any{
                "site":  "example.com",
                "event": "site_publish",
            },
            signature:      "invalid-signature",
            expectedStatus: http.StatusUnauthorized,
        },
    }
}

Receiving Webhooks

Endpoint Requirements

Your webhook endpoint must:

Accept POST requests with Content-Type: application/json
Return 200-299 status codes within 5 seconds
Validate the signature before processing
Be idempotent - handle duplicate deliveries gracefully

Example Handler (Go)

func handleAdaptWebhook(w http.ResponseWriter, r *http.Request) {
    // Read body
    body, err := io.ReadAll(r.Body)
    if err != nil {
        http.Error(w, "Failed to read body", http.StatusBadRequest)
        return
    }
    
    // Get signature from payload
    var payload map[string]any
    if err := json.Unmarshal(body, &payload); err != nil {
        http.Error(w, "Invalid JSON", http.StatusBadRequest)
        return
    }
    
    signature, ok := payload["signature"].(string)
    if !ok {
        http.Error(w, "Missing signature", http.StatusUnauthorized)
        return
    }
    
    // Validate signature
    secret := os.Getenv("ADAPT_WEBHOOK_SECRET")
    if !validateWebhook(body, signature, secret) {
        http.Error(w, "Invalid signature", http.StatusUnauthorized)
        return
    }
    
    // Process webhook
    event := payload["event"].(string)
    switch event {
    case "job.completed":
        handleJobCompleted(payload["data"])
    case "job.failed":
        handleJobFailed(payload["data"])
    case "job.progress":
        handleJobProgress(payload["data"])
    }
    
    // Return success
    w.WriteHeader(http.StatusOK)
    json.NewEncoder(w).Encode(map[string]string{"status": "received"})
}

Example Handler (Python/Flask)

from flask import Flask, request, jsonify
import json

app = Flask(__name__)

@app.route('/webhooks/adapt', methods=['POST'])
def handle_adapt_webhook():
    # Read body
    body = request.get_data()
    
    # Parse JSON
    payload = request.get_json()
    if not payload:
        return jsonify({"error": "Invalid JSON"}), 400
    
    # Validate signature
    signature = payload.get('signature')
    if not signature:
        return jsonify({"error": "Missing signature"}), 401
    
    secret = os.environ['ADAPT_WEBHOOK_SECRET']
    if not validate_webhook(body, signature, secret):
        return jsonify({"error": "Invalid signature"}), 401
    
    # Process webhook
    event = payload['event']
    if event == 'job.completed':
        handle_job_completed(payload['data'])
    elif event == 'job.failed':
        handle_job_failed(payload['data'])
    elif event == 'job.progress':
        handle_job_progress(payload['data'])
    
    return jsonify({"status": "received"}), 200

Retry Logic

Adapt implements automatic retries for failed webhook deliveries:

Initial Attempt

Webhook sent immediately when event occurs.

First Retry

After 1 minute if initial delivery fails.

Second Retry

After 5 minutes (exponential backoff).

Third Retry

After 15 minutes (final attempt).

Give Up

After 3 failed attempts, webhook is marked as failed.

Failure Conditions:

HTTP status code >= 500
Network timeout (5 seconds)
Connection refused
TLS errors

4xx status codes (except 429) are not retried, as they indicate client errors that won’t be resolved by retrying.

Idempotency

Handling Duplicates

From internal/api/webhook_test.go:211-295, Adapt’s webhook system is designed to be idempotent:

func TestWebhookDuplication(t *testing.T) {
    processedWebhooks := make(map[string]bool)
    
    tests := []struct {
        webhookID      string
        expectedStatus int
    }{
        {"webhook-123", http.StatusOK},     // First
        {"webhook-123", http.StatusOK},     // Duplicate
        {"webhook-456", http.StatusOK},     // Different
    }
    
    // Handler checks if webhook already processed
    if _, exists := processedWebhooks[webhookID]; exists {
        // Return success but don't reprocess
        return
    }
    
    processedWebhooks[webhookID] = true
}

Your Implementation: Store processed webhook IDs (from data.job_id + timestamp) to detect duplicates:

type WebhookLog struct {
    ID        string
    Event     string
    JobID     string
    Timestamp time.Time
    Processed bool
}

// Check if already processed
webhookID := fmt.Sprintf("%s:%s", payload["data"].(map[string]any)["job_id"], payload["timestamp"])
if db.WebhookExists(webhookID) {
    return // Already processed
}

// Process and mark as complete
db.CreateWebhookLog(webhookID, event, jobID)

Rate Limits

From internal/api/webhook_test.go:368-451, Adapt respects rate limits:

Max requests: 100 per minute per endpoint
Burst capacity: 10 requests
Headers returned:
- X-RateLimit-Limit: 100
- X-RateLimit-Remaining: 95
- Retry-After: 60 (when limited)

If your endpoint returns 429 (Too Many Requests), Adapt will respect the Retry-After header and delay the next attempt.

Testing

Local Testing with ngrok

Expose your local endpoint to receive webhooks:

# Install ngrok
brew install ngrok  # macOS

# Start tunnel
ngrok http 3000

# Use the HTTPS URL as your webhook endpoint
https://abc123.ngrok.io/webhooks/adapt

Test Payload

Send a test webhook to your endpoint:

curl -X POST https://your-domain.com/webhooks/adapt \
  -H "Content-Type: application/json" \
  -d '{
    "event": "job.completed",
    "timestamp": "2024-03-03T14:45:00Z",
    "data": {
      "job_id": "job_test123",
      "organisation_id": "org_test",
      "domain": "example.com",
      "status": "completed",
      "total_tasks": 10,
      "completed_tasks": 10
    },
    "signature": "sha256=calculated_signature"
  }'

Common Issues

Signature Validation Fails

Symptom: All webhooks rejected with 401 Causes:

Using wrong secret for validation
Modifying payload before validation
Not using constant-time comparison
Character encoding mismatch

Solution: Validate signature against raw request body before parsing JSON

Duplicate Events

Symptom: Same webhook received multiple times Cause: Retry logic after transient failures Solution: Implement idempotency checking using webhook ID + timestamp

Timeout Errors

Symptom: Webhooks fail despite endpoint working Cause: Response time > 5 seconds Solution: Return 200 OK immediately, process webhook asynchronously

func handleWebhook(w http.ResponseWriter, r *http.Request) {
    // Validate and read payload
    body, _ := io.ReadAll(r.Body)
    
    // Return success immediately
    w.WriteHeader(http.StatusOK)
    json.NewEncoder(w).Encode(map[string]string{"status": "received"})
    
    // Process asynchronously
    go func() {
        processWebhookAsync(body)
    }()
}

Webhook Registration (Coming Soon)

In the future, you’ll be able to register custom webhooks via the API:

POST /v1/webhooks
Authorization: Bearer <your_jwt_token>
Content-Type: application/json

{
  "url": "https://your-domain.com/webhooks/adapt",
  "events": ["job.completed", "job.failed"],
  "secret": "your-webhook-secret"
}

Best Practices

Always Validate Signatures

Verify HMAC signature before processing any webhook.

Respond Quickly

Return 200 OK within 5 seconds. Process webhooks asynchronously.

Handle Duplicates

Implement idempotency checks using webhook ID.

Monitor Failures

Log failed webhook deliveries and alert on repeated failures.

Use HTTPS

Always use HTTPS endpoints to protect webhook secrets in transit.

Next Steps

Webflow Integration

Learn about incoming Webflow webhooks

API Reference

Explore the full API documentation

Get Started

Core Features

Integrations

Guides

Custom Webhooks

Overview

Webhook Events

Job Completed

Job Failed

Job Progress

Security

HMAC Signature Validation

Test Signature Validation

Receiving Webhooks

Endpoint Requirements

Example Handler (Go)

Example Handler (Python/Flask)

Retry Logic

Idempotency

Handling Duplicates

Rate Limits

Testing

Local Testing with ngrok

Test Payload

Common Issues

Signature Validation Fails

Duplicate Events

Timeout Errors

Webhook Registration (Coming Soon)

Best Practices

Next Steps

Webflow Integration

API Reference

Build docs developers (and LLMs) love

Get Started

Core Features

Integrations

Guides

​Overview

​Webhook Events

​Job Completed

​Job Failed

​Job Progress

​Security

​HMAC Signature Validation

​Test Signature Validation

​Receiving Webhooks

​Endpoint Requirements

​Example Handler (Go)

​Example Handler (Python/Flask)

​Retry Logic

​Idempotency

​Handling Duplicates

​Rate Limits

​Testing

​Local Testing with ngrok

​Test Payload

​Common Issues

​Signature Validation Fails

​Duplicate Events

​Timeout Errors

​Webhook Registration (Coming Soon)

​Best Practices

​Next Steps

Webflow Integration

API Reference

Build docs developers (and LLMs) love

Overview

Webhook Events

Job Completed

Job Failed

Job Progress

Security

HMAC Signature Validation

Test Signature Validation

Receiving Webhooks

Endpoint Requirements

Example Handler (Go)

Example Handler (Python/Flask)

Retry Logic

Idempotency

Handling Duplicates

Rate Limits

Testing

Local Testing with ngrok

Test Payload

Common Issues

Signature Validation Fails

Duplicate Events

Timeout Errors

Webhook Registration (Coming Soon)

Best Practices

Next Steps