Batch Evaluation

​

Overview

• Normalize large datasets of social media claims
• Compare multiple AI models on the same dataset
• Evaluate normalization quality with automated metrics
• Export results for further analysis
• Test different prompting strategies at scale

​

Supported Features

​

Dataset Formats

id,claim,reference_normalization
1,"OMG the president is lying about the economy!!!","The president made claims about economic performance"
2,"Scientists say coffee causes cancer now???","Some studies suggest a link between coffee consumption and cancer risk"

{"id": 1, "claim": "OMG the president is lying about the economy!!!", "reference": "The president made claims about economic performance"}
{"id": 2, "claim": "Scientists say coffee causes cancer now???", "reference": "Some studies suggest a link between coffee consumption and cancer risk"}

​

Required Fields

• id: Unique identifier for each claim
• claim: The noisy, unstructured social media post to normalize
• reference (optional): Ground truth normalization for evaluation metrics

​

Evaluation Metrics

• Primary metric for claim normalization quality
• Measures semantic similarity between generated and reference normalizations
• Accounts for synonyms, stemming, and paraphrasing
• Score range: 0.0 (poor) to 1.0 (perfect match)

• Verifiability: How easily the normalized claim can be fact-checked
• Check-Worthiness: Importance and urgency of verifying this claim
• Factual Consistency: Accuracy without distortion or hallucination
• Clarity: Understandability and lack of ambiguity
• Relevance: Connection to current events and public discourse

​

Batch Evaluation Workflow

id,claim,reference
1,"Your noisy claim here","Reference normalization (optional)"
2,"Another claim...","Another reference..."

{
  "type": "progress",
  "processed": 15,
  "total": 100,
  "current_claim": "Scientists say coffee causes cancer",
  "model": "gpt-4o-mini"
}

id,original_claim,normalized_claim,meteor_score,verifiability,model
1,"OMG the president...","The president made claims...",0.85,0.92,gpt-4o

[
  {
    "id": 1,
    "original_claim": "OMG the president...",
    "normalized_claim": "The president made claims...",
    "meteor_score": 0.85,
    "metrics": {
      "verifiability": 0.92,
      "check_worthiness": 0.88,
      "clarity": 0.95
    },
    "model": "gpt-4o"
  }
]

​

Understanding Evaluation Metrics

​

METEOR Scoring

• 0.0 - 0.3: Poor normalization, significant semantic drift
• 0.3 - 0.5: Moderate quality, captures some key information
• 0.5 - 0.7: Good normalization, preserves main claims
• 0.7 - 0.9: Excellent normalization, high semantic similarity
• 0.9 - 1.0: Near-perfect match with reference

• Unigram matching between normalized and reference text
• Synonym and stemming alignment
• Word order preservation
• Penalty for fragmentation

​

DeepEval Metrics

• Contains specific, factual assertions
• Evidence can be found to support/refute
• Not overly vague or opinion-based
• Includes relevant time/location context

• Potential harm if false
• Reach and influence potential
• Public interest level
• Impact on vulnerable populations

• No hallucinated information
• Maintains original context
• Doesn’t misrepresent source
• Avoids introducing new claims

• Clear, simple language
• Lacks ambiguous terms
• Self-contained statement
• Concise yet comprehensive

​

Model Comparison

• Average METEOR score per model
• Consistency across claims
• Processing time per model
• Cost per model (based on token usage)

​

API Integration

import requests

# Upload dataset
files = {'file': open('claims.csv', 'rb')}
response = requests.post(
    'https://api.checkthat.ai/batch/upload',
    files=files,
    headers={'Authorization': f'Bearer {api_key}'}
)

batch_id = response.json()['batch_id']

# Start evaluation
config = {
    'models': ['gpt-4o', 'claude-opus-4'],
    'strategy': 'zero-shot',
    'metrics': ['meteor', 'verifiability']
}
requests.post(
    f'https://api.checkthat.ai/batch/{batch_id}/evaluate',
    json=config,
    headers={'Authorization': f'Bearer {api_key}'}
)

# Monitor via WebSocket
import websocket

ws = websocket.create_connection(
    f'wss://api.checkthat.ai/batch/{batch_id}/progress'
)

while True:
    result = ws.recv()
    print(result)
    if 'completed' in result:
        break

​

Best Practices

​

Dataset Preparation

1. Clean your data: Remove duplicates and invalid entries
2. Balance your dataset: Include diverse claim types
3. Add quality references: Better references = more accurate metrics
4. Test with small batches: Validate with 10-20 claims first

​

Model Selection

1. Start with mid-tier models: Test with GPT-4o-mini or Claude Sonnet
2. Use premium for final runs: Reserve GPT-5/Opus 4.1 for production
3. Compare 2-3 models: Find the best fit for your use case
4. Consider cost vs quality: Balance budget with accuracy needs

​

Optimization

1. Batch size: Keep batches under 1000 claims for manageable processing
2. Concurrent processing: The system automatically parallelizes when possible
3. Error handling: Failed claims are retried automatically (up to 3 attempts)
4. Rate limits: Respect API rate limits by spacing requests

​

Troubleshooting

​

Upload Errors

• Ensure proper CSV structure with headers
• Check for unescaped quotes or commas
• Verify UTF-8 encoding

• Split dataset into multiple files (max 10MB each)
• Remove unnecessary columns
• Use JSONL for more efficient storage

​

Processing Issues

• Extremely long claims may timeout
• Break very long posts into multiple claims
• Consider using a more powerful model

• Check if references are too different from your normalization style
• Verify references are actual normalizations, not original claims
• Consider if your prompting strategy needs adjustment

​

Next Steps

• Learn about [Claim Normalization Strategies(/guides/prompting-strategies)
• Explore [Evaluation Metrics(/concepts/evaluation-metrics) in detail
• Try Interactive Chat for single claim testing
• Review [API Documentation(/api/overview) for programmatic access