Semantic Validator

The SemanticValidator enforces epistemic type contracts on model outputs, ensuring responses conform to declared AXON semantic types.

Model Output → [Validator] → ValidationResult (pass/fail + violations)

What It Validates

The validator enforces five categories of semantic rules:

Type Category Matching — FactualClaim ≠ Opinion
Confidence Floor Enforcement — confidence >= threshold
Structured Field Presence — Required fields exist
Range Validation — RiskScore ∈ [0.0, 1.0]
Epistemic Exclusion — Mutually exclusive types

Key Principle: The validator never modifies output — it only observes and judges.

Implementation

from dataclasses import dataclass
from typing import Any

class SemanticValidator:
    """Validates model outputs against declared AXON semantic types."""

    def __init__(
        self,
        custom_types: dict[str, list[str]] | None = None,
    ) -> None:
        self._custom_types = custom_types or {}

    def validate(
        self,
        output: Any,
        expected_type: str = "",
        confidence_floor: float | None = None,
        type_fields: list[str] | None = None,
        range_min: float | None = None,
        range_max: float | None = None,
        tracer: Tracer | None = None,
        step_name: str = "",
    ) -> ValidationResult:
        """Run all applicable validation checks on the output."""
        violations: list[Violation] = []
        extracted_confidence: float | None = None

        # Check 1: Type category
        if expected_type:
            type_violations = self._validate_type_category(output, expected_type)
            violations.extend(type_violations)

        # Check 2: Confidence floor
        if confidence_floor is not None:
            conf_result = self._validate_confidence(
                output, confidence_floor, tracer, step_name
            )
            violations.extend(conf_result[0])
            extracted_confidence = conf_result[1]

        # Check 3: Structured fields
        if type_fields:
            field_violations = self._validate_fields(output, type_fields)
            violations.extend(field_violations)

        # Check 4: Range validation
        if range_min is not None or range_max is not None:
            range_violations = self._validate_range(output, range_min, range_max)
            violations.extend(range_violations)

        has_errors = any(v.severity == "error" for v in violations)
        return ValidationResult(
            is_valid=not has_errors,
            violations=tuple(violations),
            confidence=extracted_confidence,
        )

Built-in Type Registry

Epistemic Types (Mutually Exclusive)

EPISTEMIC_TYPES = frozenset({
    "FactualClaim",  # Verifiable, objective
    "Opinion",       # Subjective judgment
    "Uncertainty",   # Explicitly uncertain
    "Speculation",   # Hypothetical
})

Rule: These types are mutually exclusive. An Opinion can never satisfy a FactualClaim.

Analysis Types (with Ranges)

ANALYSIS_TYPES = frozenset({
    "RiskScore",
    "ConfidenceScore",
    "SentimentScore",
})

RANGED_TYPE_BOUNDS: dict[str, tuple[float, float]] = {
    "RiskScore": (0.0, 1.0),
    "ConfidenceScore": (0.0, 1.0),
    "SentimentScore": (-1.0, 1.0),
}

Validation Rules

1. Type Category Matching

Rule: Output must match the expected semantic type category.

def _validate_type_category(
    self, output: Any, expected_type: str
) -> list[Violation]:
    violations: list[Violation] = []

    # For dict outputs, check declared type field
    if isinstance(output, dict):
        declared = output.get("type", output.get("_type", ""))
        if declared and declared != expected_type:
            # Epistemic exclusion check
            if (
                expected_type in EPISTEMIC_TYPES
                and declared in EPISTEMIC_TYPES
                and declared != expected_type
            ):
                violations.append(
                    Violation(
                        rule="epistemic_exclusion",
                        message=(
                            f"Epistemic type mismatch: expected '{expected_type}' "
                            f"but output declares '{declared}'. "
                            f"These types are mutually exclusive."
                        ),
                        expected=expected_type,
                        actual=declared,
                    )
                )

    return violations

Example Failure:

output = {"type": "Opinion", "content": "I think..."}
expected_type = "FactualClaim"

# Violation: Epistemic type mismatch

2. Confidence Floor Enforcement

Rule: Confidence score must meet the minimum threshold.

def _validate_confidence(
    self,
    output: Any,
    floor: float,
    tracer: Tracer | None,
    step_name: str,
) -> tuple[list[Violation], float | None]:
    violations: list[Violation] = []
    extracted: float | None = None

    # Try to extract confidence from output
    if isinstance(output, dict):
        raw = output.get("confidence", output.get("_confidence"))
        if raw is not None:
            try:
                extracted = float(raw)
            except (TypeError, ValueError):
                pass

    if extracted is not None:
        passed = extracted >= floor

        if tracer:
            tracer.emit_confidence_check(
                step_name=step_name,
                score=extracted,
                floor=floor,
                passed=passed,
            )

        if not passed:
            violations.append(
                Violation(
                    rule="confidence_floor",
                    message=(
                        f"Confidence {extracted:.2f} is below the "
                        f"floor of {floor:.2f}."
                    ),
                    expected=f">= {floor}",
                    actual=f"{extracted:.2f}",
                )
            )

    return violations, extracted

Example:

output = {"confidence": 0.72, "content": "..."}
confidence_floor = 0.85

# Violation: Confidence 0.72 is below the floor of 0.85

3. Structured Field Presence

Rule: All required fields must exist in structured output.

def _validate_fields(
    self, output: Any, required_fields: list[str]
) -> list[Violation]:
    violations: list[Violation] = []

    if not isinstance(output, dict):
        violations.append(
            Violation(
                rule="structured_type",
                message=(
                    f"Expected structured output (dict) with fields {required_fields}, "
                    f"but got {type(output).__name__}."
                ),
                expected="dict",
                actual=type(output).__name__,
            )
        )
        return violations

    missing = [f for f in required_fields if f not in output]
    if missing:
        violations.append(
            Violation(
                rule="missing_fields",
                message=(
                    f"Missing required fields: {missing}. "
                    f"Present fields: {list(output.keys())}."
                ),
                expected=str(required_fields),
                actual=str(list(output.keys())),
            )
        )

    return violations

Example:

output = {"parties": "Acme Corp", "date": "2024-01-15"}
required_fields = ["parties", "date", "termination_clause"]

# Violation: Missing required fields: ['termination_clause']

4. Range Validation

Rule: Numeric values must fall within declared ranges.

def _validate_range(
    self,
    output: Any,
    range_min: float | None,
    range_max: float | None,
) -> list[Violation]:
    violations: list[Violation] = []

    # Extract numeric value
    value: float | None = None
    if isinstance(output, (int, float)):
        value = float(output)
    elif isinstance(output, dict):
        raw = output.get("value", output.get("score"))
        if raw is not None:
            try:
                value = float(raw)
            except (TypeError, ValueError):
                pass

    if value is None:
        return violations

    if range_min is not None and value < range_min:
        violations.append(
            Violation(
                rule="range_below_min",
                message=f"Value {value} is below minimum {range_min}.",
                expected=f">= {range_min}",
                actual=str(value),
            )
        )

    if range_max is not None and value > range_max:
        violations.append(
            Violation(
                rule="range_above_max",
                message=f"Value {value} exceeds maximum {range_max}.",
                expected=f"<= {range_max}",
                actual=str(value),
            )
        )

    return violations

Example:

output = {"score": 1.3}
range_max = 1.0

# Violation: Value 1.3 exceeds maximum 1.0

Violation Structure

@dataclass(frozen=True)
class Violation:
    """A single validation failure."""
    rule: str          # Which rule failed
    message: str       # Human-readable description
    expected: str = "" # What was expected
    actual: str = ""   # What was found
    severity: str = "error"  # "error" or "warning"

Example:

Violation(
    rule="confidence_floor",
    message="Confidence 0.72 is below the floor of 0.85.",
    expected=">= 0.85",
    actual="0.72",
    severity="error"
)

ValidationResult

@dataclass(frozen=True)
class ValidationResult:
    """The aggregate outcome of a validation pass."""
    is_valid: bool = True
    violations: tuple[Violation, ...] = ()
    confidence: float | None = None

    @property
    def errors(self) -> list[Violation]:
        return [v for v in self.violations if v.severity == "error"]

    @property
    def warnings(self) -> list[Violation]:
        return [v for v in self.violations if v.severity == "warning"]

Usage:

result = validator.validate(
    output=model_response,
    expected_type="FactualClaim",
    confidence_floor=0.85,
)

if not result.is_valid:
    for violation in result.errors:
        print(f"Error: {violation.message}")

Integration with Executor

The Executor calls the validator after anchor checking:

def _validate_response_cps(
    self,
    response: ModelResponse,
    step: CompiledStep,
    step_name: str,
    tracer: Tracer,
    on_success: Callable[[ValidationResult], Any],
    on_failure: Callable[[list[str]], Any],
) -> Any:
    result = self._validator.validate(
        output=response.structured or response.content,
        expected_type=step.metadata.get("output_type", ""),
        confidence_floor=step.metadata.get("confidence_floor"),
    )

    tracer.emit_validation_result(
        step_name=step_name,
        passed=result.is_valid,
        violations=[v.message for v in result.violations],
    )

    if not result.is_valid:
        return on_failure([v.message for v in result.violations])

    return on_success(result)

Custom Types

The validator supports user-defined types with custom field requirements:

custom_types = {
    "ContractAnalysis": ["parties", "effective_date", "risk_score"],
    "LegalOpinion": ["conclusion", "reasoning", "precedents"],
}

validator = SemanticValidator(custom_types=custom_types)

result = validator.validate(
    output=model_response,
    expected_type="ContractAnalysis",
)

# Automatically validates presence of: parties, effective_date, risk_score

Validation + Retry

When validation fails, the RetryEngine injects failure context:

# Attempt 1: Model output
{"confidence": 0.72, "content": "..."}

# Validation fails: confidence too low

# Attempt 2: Model receives failure context
failure_context = "Previous response had confidence 0.72 below floor 0.85. Please increase confidence or express uncertainty."

# Model adjusts:
{"confidence": 0.88, "content": "..."}

See Retry Engine for details.

Trace Integration

Validation results are fully traced:

tracer.emit_validation_result(
    step_name="Extract",
    passed=False,
    expected_type="EntityMap",
    violations=[
        "Missing required fields: ['termination_clause']",
        "Confidence 0.72 is below the floor of 0.85",
    ],
)

See Tracer for observability details.

Next Steps

Retry Engine

See how validation failures trigger adaptive retry

Type Checker

Compare with compile-time epistemic validation

Tracer

Understand execution observability

Compiler

Backends

Runtime

Semantic Validator

Semantic Validator

What It Validates

Implementation

Built-in Type Registry

Epistemic Types (Mutually Exclusive)

Analysis Types (with Ranges)

Validation Rules

1. Type Category Matching

2. Confidence Floor Enforcement

3. Structured Field Presence

4. Range Validation

Violation Structure

ValidationResult

Integration with Executor

Custom Types

Validation + Retry

Trace Integration

Next Steps

Retry Engine

Type Checker

Tracer

Build docs developers (and LLMs) love

Compiler

Backends

Runtime

​Semantic Validator

​What It Validates

​Implementation

​Built-in Type Registry

​Epistemic Types (Mutually Exclusive)

​Analysis Types (with Ranges)

​Validation Rules

​1. Type Category Matching

​2. Confidence Floor Enforcement

​3. Structured Field Presence

​4. Range Validation

​Violation Structure

​ValidationResult

​Integration with Executor

​Custom Types

​Validation + Retry

​Trace Integration

​Next Steps

Retry Engine

Type Checker

Tracer

Build docs developers (and LLMs) love

Semantic Validator

What It Validates

Implementation

Built-in Type Registry

Epistemic Types (Mutually Exclusive)

Analysis Types (with Ranges)

Validation Rules

1. Type Category Matching

2. Confidence Floor Enforcement

3. Structured Field Presence

4. Range Validation

Violation Structure

ValidationResult

Integration with Executor

Custom Types

Validation + Retry

Trace Integration

Next Steps