POST /api/admin/ingestion/run

Overview

The ingestion API triggers TechCal’s automated event collection pipeline. It fetches events from configured sources (RSS feeds, APIs, ICS calendars, HTML scrapers), normalizes data, and queues events for moderation.

Admin-only endpoint. Requires is_admin=true in user profile.

Authentication

Requires both:

Valid Supabase authentication token
Admin user privileges (profiles.is_admin = true)

Authorization: Bearer <supabase_access_token>

Request

HTTP Method

POST /api/admin/ingestion/run

Headers

Content-Type

string

required

Must be application/json

Body Parameters

sourceId

string

UUID of specific ingestion source to run. If omitted, runs all active sources.

limit

number

default:"100"

Maximum number of events to normalize after ingestion

Response

Success Response (200)

success

boolean

required

Always true for successful ingestion runs

message

string

required

Human-readable status message

summary

object

required

Show summary properties

ingestion

object

required

Ingestion pipeline results

totalSources

number

Number of sources processed

successful

number

Sources that completed successfully

failed

number

Sources that failed

totalEventsFetched

number

Raw events fetched from sources

totalRecordsQueued

number

Events queued for normalization

totalErrors

number

Total errors across all sources

normalization

object

required

Normalization pipeline results

processed

number

Events processed by normalizer

succeeded

number

Events successfully normalized

failed

number

Events that failed normalization

errors

string[]

Error messages from normalization

results

object[]

required

Per-source detailed results

sourceId

string

Source UUID

sourceName

string

Human-readable source name

success

boolean

Whether this source completed successfully

eventsFetched

number

Raw events fetched

recordsQueued

number

Events queued for processing

errors

number

Error count for this source

errorMessages

string[]

Specific error messages

timestamp

string

required

ISO 8601 timestamp of ingestion completion

Error Responses

{
  "error": "Unauthorized"
}

Examples

Run All Sources

curl -X POST https://kurecal.app/api/admin/ingestion/run \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <admin_token>" \
  -d '{}'

Run Specific Source

curl -X POST https://kurecal.app/api/admin/ingestion/run \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <admin_token>" \
  -d '{
    "sourceId": "550e8400-e29b-41d4-a716-446655440000",
    "limit": 50
  }'

Ingestion Pipeline

Pipeline Stages

Source Selection
- If sourceId provided: Fetch single source
- Otherwise: Fetch all sources where is_active=true
Interval Check
- Skip sources fetched within their fetch_interval_minutes
- Manual triggers bypass this check (force fetch)
Data Collection
- RSS: Parse XML feed
- API: Call REST endpoint
- ICS: Parse iCalendar file
- HTML: Scrape event listings
Deduplication
- Check for existing events by URL
- Use fuzzy matching for title/date similarity
- Skip duplicates, queue new events
Normalization
- Parse dates to ISO 8601
- Extract location and format
- Normalize tags and categories
- Calculate quality score
Quality Control
- Auto-publish: Score ≥ 75% → Publish immediately
- Moderation queue: Score < 75% → Requires review
- Auto-reject: Score < 50% → Flag for manual review

Quality Scoring

Events receive a quality score (0-100) based on:

Factor	Weight	Criteria
Title Quality	20%	Length, clarity, no spam patterns
Description Quality	20%	Length, formatting, detail level
Date Validity	15%	Valid dates, reasonable duration
Location Data	15%	Complete location or virtual indicator
Speaker Info	10%	Speaker names and URLs present
Organizer Info	10%	Reputable organizer
Image Quality	5%	High-res image present
Tag Relevance	5%	Relevant tech tags

Supported Source Types

RSS Feed
API
ICS Calendar
HTML Scraper

Configuration:

{
  "type": "rss",
  "url": "https://example.com/events.rss",
  "fetch_interval_minutes": 60
}

Extraction:

Title: <title>
Description: <description>
URL: <link>
Date: <pubDate> or <dc:date>

Configuration:

{
  "type": "api",
  "url": "https://api.example.com/events",
  "method": "GET",
  "headers": { "Authorization": "Bearer ..." },
  "fetch_interval_minutes": 120
}

Extraction:

Parse JSON response
Map fields via configuration

Configuration:

{
  "type": "ics",
  "url": "https://example.com/events.ics",
  "fetch_interval_minutes": 360
}

Extraction:

SUMMARY → title
DESCRIPTION → description
DTSTART → start_time
DTEND → end_time
LOCATION → location

Configuration:

{
  "type": "html",
  "url": "https://example.com/events",
  "selectors": {
    "container": ".event-card",
    "title": "h3.title",
    "date": ".date",
    "url": "a.link"
  },
  "fetch_interval_minutes": 1440
}

Extraction:

Use CSS selectors
Parse structured data

Cron Setup

Vercel Cron Configuration

Configured in vercel.json:

{
  "crons": [
    {
      "path": "/api/admin/ingestion/cron",
      "schedule": "0 * * * *"
    }
  ]
}

Cron Authentication

The cron endpoint requires a secret header:

Authorization: Bearer <CRON_SECRET>

Generate secret:

openssl rand -hex 32

Set in environment:

CRON_SECRET=your_generated_secret

Manual vs Cron Execution

Aspect	Manual (`/run`)	Cron (`/cron`)
Auth	Admin user token	CRON_SECRET header
Interval check	Bypassed (force)	Enforced
Error handling	Returns 500	Logs to Sentry
Timeout	5 minutes	60 seconds

Moderation Workflow

Moderation Queue

Events requiring review:

SELECT * FROM ingestion_queue
WHERE status = 'pending'
  AND quality_score < 75
ORDER BY quality_score DESC, created_at DESC;

Admin Dashboard

Manage queued events at /admin/ingestion/moderation:

Approve: Publish event immediately
Reject: Mark as spam/duplicate
Edit & Approve: Fix issues then publish
Bulk Actions: Approve/reject multiple events

Auto-Moderation Rules

Certain events are auto-moderated:

if (event.quality_score >= 75 && event.organizer_reputation === 'verified') {
  await publishEvent(event);
} else if (event.quality_score < 50) {
  await rejectEvent(event, 'Low quality score');
} else {
  await queueForModeration(event);
}

Troubleshooting

Common Issues

“Service role credentials not configured”

Set SUPABASE_SERVICE_ROLE_KEY environment variable
Restart application after setting

“Admin access required”

Run: UPDATE profiles SET is_admin = TRUE WHERE id = '<your_user_id>'
Get user ID from Supabase Dashboard → Authentication → Users

“Source fetch interval not elapsed”

Manual triggers bypass this automatically
Reduce fetch_interval_minutes in source config

“Normalization failed: Invalid date format”

Check source date format matches expected pattern
Add custom date parser in normalization config

Debug Mode

Enable detailed logging:

NEXT_PUBLIC_LOG_LEVEL=debug

Logs include:

Source fetch URLs and responses
Deduplication match details
Normalization field mappings
Quality score calculations

Event Ingestion Setup Guide - Full setup instructions
Admin Dashboard - Web UI for managing ingestion
Moderation Queue - Review pending events

Events API

User & Profile

Services

POST /api/admin/ingestion/run

Overview

Authentication

Request

HTTP Method

Headers

Body Parameters

Response

Success Response (200)

Error Responses

Examples

Run All Sources

Run Specific Source

Ingestion Pipeline

Pipeline Stages

Quality Scoring

Supported Source Types

Cron Setup

Vercel Cron Configuration

Cron Authentication

Manual vs Cron Execution

Moderation Workflow

Moderation Queue

Admin Dashboard

Auto-Moderation Rules

Troubleshooting

Common Issues

Debug Mode

Build docs developers (and LLMs) love

Events API

User & Profile

Services

​Overview

​Authentication

​Request

​HTTP Method

​Headers

​Body Parameters

​Response

​Success Response (200)

​Error Responses

​Examples

​Run All Sources

​Run Specific Source

​Ingestion Pipeline

​Pipeline Stages

​Quality Scoring

​Supported Source Types

​Cron Setup

​Vercel Cron Configuration

​Cron Authentication

​Manual vs Cron Execution

​Moderation Workflow

​Moderation Queue

​Admin Dashboard

​Auto-Moderation Rules

​Troubleshooting

​Common Issues

​Debug Mode

​Related Documentation

Build docs developers (and LLMs) love

Overview

Authentication

Request

HTTP Method

Headers

Body Parameters

Response

Success Response (200)

Error Responses

Examples

Run All Sources

Run Specific Source

Ingestion Pipeline

Pipeline Stages

Quality Scoring

Supported Source Types

Cron Setup

Vercel Cron Configuration

Cron Authentication

Manual vs Cron Execution

Moderation Workflow

Moderation Queue

Admin Dashboard

Auto-Moderation Rules

Troubleshooting

Common Issues

Debug Mode

Related Documentation