Scoring Module

Overview

The Scoring module implements Dream Foundry’s three-criteria evaluation system:

Success - Did it produce valid output without errors?
Quality - Does the output meet strict content requirements?
Speed - How fast did it execute?

Scores are weighted (Success 20%, Quality 60%, Speed 20%) to produce a final total score.

Core Function

score_candidate()

Scores a candidate across all three criteria and returns a comprehensive result.

candidate_id

str

required

Unique identifier for the candidate being scored

produced_output

bool

required

Whether the candidate produced any output file

error_occurred

bool

required

Whether any errors occurred during execution

artifact_content

str | None

required

The content of the artifact produced by the candidate. Can be None if no output was generated.

runtime_seconds

float

required

Execution time in seconds

error_message

str | None

Optional error message if execution failed

Returns: ScoreResult - Complete scoring breakdown with success, quality, and speed metrics

from src.scoring import score_candidate

# Score a successful candidate
result = score_candidate(
    candidate_id="alpha",
    produced_output=True,
    error_occurred=False,
    artifact_content="**Event 1**\nDate: January 24, 2026...",
    runtime_seconds=5.2
)

print(f"Total Score: {result.total_score}")
print(f"Quality: {result.quality_score}")
print(f"Speed: {result.speed_score}")

Data Classes

ScoreResult

Complete scoring result for a candidate.

candidate_id

str

Identifier of the scored candidate

success

bool

Overall success status (produced output AND no errors)

produced_output

bool

Whether the candidate produced any output

error_occurred

bool

Whether any errors occurred during execution

error_message

str | None

Error message if execution failed

quality

QualityResult | None

Detailed quality validation results. None if no artifact was produced.

quality_score

float

Quality score from 0-100

runtime_seconds

float

Execution time in seconds

speed_score

float

Speed score from 0-100 (faster = higher)

total_score

float

Weighted total score combining all criteria

QualityResult

Detailed quality validation result.

passed

bool

Whether the output passed all quality checks

total_events

int

Total number of events found in the output

valid_events

int

Number of events that passed all validation criteria

has_daytona_event

bool

Whether a hackathon event was found (required)

invalid_date_events

list

List of events with dates outside January 24-31, 2026

invalid_location_events

list

List of events outside SF/San Jose/Palo Alto/Mountain View

non_ai_events

list

List of events not AI-related

invalid_urls

list

List of events with invalid or missing URLs

score

float

Quality score from 0-100

details

str

Human-readable summary of quality issues

from src.scoring import score_candidate

result = score_candidate(
    candidate_id="beta",
    produced_output=True,
    error_occurred=False,
    artifact_content=markdown_content,
    runtime_seconds=12.4
)

if result.quality:
    q = result.quality
    print(f"Events: {q.valid_events}/{q.total_events} valid")
    print(f"Hackathon event: {'Yes' if q.has_daytona_event else 'No'}")
    print(f"Quality passed: {q.passed}")
    
    if not q.passed:
        print(f"Issues: {q.details}")

Quality Validation

validate_quality()

Validates markdown content against strict quality criteria.

markdown_content

str

required

The markdown content to validate (typically a Discord post with AI events)

Returns: QualityResult with detailed validation breakdown

Strict Quality Criteria

All events must meet these requirements:

DATE: Must be January 24-31, 2026 ONLYLOCATION: Must be San Francisco, San Jose, Palo Alto, or Mountain View ONLYAI RELATED: Must contain AI/ML keywordsVALID URL: Must be from known event platforms (lu.ma, meetup.com, etc.)HACKATHON: At least one hackathon event MUST be included

Valid Date Range

VALID_DATE_START = datetime(2026, 1, 24)
VALID_DATE_END = datetime(2026, 1, 31)

Supported date formats:

"Saturday, January 24, 2026"
"January 24, 2026"
"2026-01-24"
"01/24/2026"

Valid Locations

VALID_LOCATIONS = [
    "san francisco",
    "san jose",
    "palo alto",
    "mountain view",
]

Location matching is case-insensitive and partial match.

Valid URL Patterns

VALID_URL_PATTERNS = [
    r'https?://(www\.)?lu\.ma/',
    r'https?://(www\.)?luma\.com/',
    r'https?://(www\.)?meetup\.com/',
    r'https?://(www\.)?eventbrite\.(com|co\.uk)/',
    r'https?://(www\.)?partiful\.com/',
    r'https?://(www\.)?supermomos\.com/',
    r'https?://(www\.)?humanx\.co/',
    r'https?://(compute\.)?daytona\.io/',
    r'https?://(www\.)?sensaihack\.com/',
]

AI Keywords

Events must contain at least one of these keywords (case-insensitive):

AI_KEYWORDS = [
    'ai', 'artificial intelligence', 'machine learning', 'ml', 'llm',
    'gpt', 'claude', 'langchain', 'rag', 'neural', 'deep learning',
    'nlp', 'natural language', 'computer vision', 'generative', 'transformer',
    'agent', 'agents', 'anthropic', 'openai', 'hugging face', 'pytorch',
    'tensorflow', 'data science', 'mlops', 'embedding', 'vector',
    'genai', 'diffusion', 'nvidia', 'google ai', 'stanford ai',
    'agi', 'humanx', 'daytona', 'compute', 'hackathon', 'tinkerers',
    'makerspace', 'founders', 'tech talk', 'deep dive', 'builders',
]

Quality Scoring Formula

Quality score (0-100) is calculated as:

score = 0.0

# Valid events: up to 40 points
score += (valid_events / total_events) * 40

# Hackathon event: 25 points
if has_hackathon_event:
    score += 25

# Event count bonus: up to 20 points (14+ events = 20 points)
score += min(20, (valid_events / 14) * 20)

# No invalid events bonus: 15 points
if no_invalid_events:
    score += 15

Passing Criteria

To pass quality validation, ALL of these must be true:

At least 6 valid events
Hackathon event included
No invalid date events
No invalid location events

from src.scoring import validate_quality

markdown = """
**Daytona AI Hackathon**
Date: January 24, 2026
Location: San Francisco
Type: Hackathon
[RSVP](https://compute.daytona.io/hackathon)

**AI Tinkerers Meetup**
Date: January 27, 2026
Location: Palo Alto
Type: Meetup
[RSVP](https://lu.ma/ai-tinkerers)
"""

result = validate_quality(markdown)

print(f"Score: {result.score}/100")
print(f"Passed: {result.passed}")
print(f"Valid Events: {result.valid_events}/{result.total_events}")
print(f"Hackathon: {result.has_daytona_event}")

Speed Scoring

calculate_speed_score()

Calculates speed score based on execution time.

runtime_seconds

float

required

Execution time in seconds

max_time

float

default:"30.0"

Maximum time threshold (scores 0 beyond this)

Returns: float - Speed score from 0-100

Speed Formula

if runtime_seconds <= 0:
    return 100.0
if runtime_seconds >= max_time:
    return 0.0
return 100 * (1 - runtime_seconds / max_time)

Examples:

0 seconds → 100.0
5 seconds → 83.3
15 seconds → 50.0
30 seconds → 0.0
45 seconds → 0.0

Total Score Calculation

calculate_total_score()

Calculates weighted total score from all criteria.

success

bool

required

Whether execution succeeded

quality_score

float

required

Quality score (0-100)

speed_score

float

required

Speed score (0-100)

weights

dict

Custom weights dictionary. Defaults to:

{'success': 0.2, 'quality': 0.6, 'speed': 0.2}

Returns: float - Total score from 0-100

Default Weights

Success: 20%
Quality: 60%
Speed: 20%

Formula

if not success:
    return 0.0

total = (
    100.0 * weights['success'] +      # Success is binary: 0 or 100
    quality_score * weights['quality'] +
    speed_score * weights['speed']
)

from src.scoring import calculate_total_score

score = calculate_total_score(
    success=True,
    quality_score=85.0,
    speed_score=75.0
)
# Result: 20.0 + 51.0 + 15.0 = 86.0

Helper Functions

parse_events_from_markdown()

Parses event data from Discord markdown format.

markdown

str

required

Markdown content containing event listings

Returns: list[dict] - List of event dictionaries Each event dictionary contains:

title: Event title
date: Date string
time: Time string (if present)
location: Location string
event_type: Event type (hackathon, meetup, etc.)
url: Event URL

from src.scoring import parse_events_from_markdown

markdown = """
**Daytona AI Hackathon**
Date: January 24, 2026
Time: 9:00 AM - 6:00 PM
Location: San Francisco
Type: Hackathon
[RSVP](https://compute.daytona.io/hackathon)
"""

events = parse_events_from_markdown(markdown)
print(events[0])
# {
#   'title': 'Daytona AI Hackathon',
#   'date': 'January 24, 2026',
#   'time': '9:00 AM - 6:00 PM',
#   'location': 'San Francisco',
#   'event_type': 'Hackathon',
#   'url': 'https://compute.daytona.io/hackathon'
# }

Validation Helper Functions

parse_date()

Parses date from various formats.

from src.scoring import parse_date
from datetime import datetime

date = parse_date("January 24, 2026")  # datetime(2026, 1, 24)

is_valid_date()

Checks if date is within valid range (Jan 24-31, 2026).

from src.scoring import is_valid_date

is_valid_date("January 24, 2026")  # True
is_valid_date("February 1, 2026")   # False

is_valid_location()

Checks if location is in allowed cities.

from src.scoring import is_valid_location

is_valid_location("San Francisco")  # True
is_valid_location("Los Angeles")    # False

is_valid_url()

Checks if URL matches valid event platform patterns.

from src.scoring import is_valid_url

is_valid_url("https://lu.ma/ai-event")      # True
is_valid_url("https://random-site.com")     # False

is_ai_related()

Checks if title contains AI keywords.

from src.scoring import is_ai_related

is_ai_related("AI Tinkerers Meetup")  # True
is_ai_related("Yoga Class")            # False

has_hackathon_event()

Checks if any event in list is a hackathon.

from src.scoring import has_hackathon_event

events = [
    {'title': 'AI Hackathon', 'event_type': 'Hackathon'},
    {'title': 'AI Meetup', 'event_type': 'Meetup'}
]

has_hackathon_event(events)  # True

Complete Example

from src.scoring import (
    score_candidate,
    validate_quality,
    parse_events_from_markdown
)

# Sample markdown output from a candidate
markdown_content = """
**Daytona AI Hackathon**
Date: January 24, 2026
Location: San Francisco
Type: Hackathon
[RSVP](https://compute.daytona.io/hackathon)

**AI Tinkerers Meetup**
Date: January 27, 2026
Location: Palo Alto
Type: Meetup
[RSVP](https://lu.ma/ai-tinkerers)
"""

# Score the candidate
result = score_candidate(
    candidate_id="gamma",
    produced_output=True,
    error_occurred=False,
    artifact_content=markdown_content,
    runtime_seconds=8.5
)

# Display results
print(f"Candidate: {result.candidate_id}")
print(f"Success: {result.success}")
print(f"Quality Score: {result.quality_score}/100")
print(f"Speed Score: {result.speed_score}/100")
print(f"Total Score: {result.total_score}/100")

if result.quality:
    print(f"\nQuality Details:")
    print(f"  Valid Events: {result.quality.valid_events}/{result.quality.total_events}")
    print(f"  Hackathon: {result.quality.has_daytona_event}")
    print(f"  Passed: {result.quality.passed}")
    print(f"  Details: {result.quality.details}")

Core Modules

Candidate Agents

Integration Modules

Overview

Core Function

score_candidate()

Data Classes

ScoreResult

QualityResult

Quality Validation

validate_quality()

Strict Quality Criteria

Valid Date Range

Valid Locations

Valid URL Patterns

AI Keywords

Quality Scoring Formula

Passing Criteria

Speed Scoring

calculate_speed_score()

Speed Formula

Total Score Calculation

calculate_total_score()

Default Weights

Formula

Helper Functions

parse_events_from_markdown()

Validation Helper Functions

parse_date()

is_valid_date()

is_valid_location()

is_valid_url()

is_ai_related()

has_hackathon_event()

Complete Example

Build docs developers (and LLMs) love

Core Modules

Candidate Agents

Integration Modules

​Overview

​Core Function

​score_candidate()

​Data Classes

​ScoreResult

​QualityResult

​Quality Validation

​validate_quality()

​Strict Quality Criteria

​Valid Date Range

​Valid Locations

​Valid URL Patterns

​AI Keywords

​Quality Scoring Formula

​Passing Criteria

​Speed Scoring

​calculate_speed_score()

​Speed Formula

​Total Score Calculation

​calculate_total_score()

​Default Weights

​Formula

​Helper Functions

​parse_events_from_markdown()

​Validation Helper Functions

​parse_date()

​is_valid_date()

​is_valid_location()

​is_valid_url()

​is_ai_related()

​has_hackathon_event()

​Complete Example

Build docs developers (and LLMs) love

Overview

Core Function

score_candidate()

Data Classes

ScoreResult

QualityResult

Quality Validation

validate_quality()

Strict Quality Criteria

Valid Date Range

Valid Locations

Valid URL Patterns

AI Keywords

Quality Scoring Formula

Passing Criteria

Speed Scoring

calculate_speed_score()

Speed Formula

Total Score Calculation

calculate_total_score()

Default Weights

Formula

Helper Functions

parse_events_from_markdown()

Validation Helper Functions

parse_date()

is_valid_date()

is_valid_location()

is_valid_url()

is_ai_related()

has_hackathon_event()

Complete Example