Skip to main content

​
What It Does

Predictor Hand is a superforecasting engine inspired by Philip Tetlock’s research. It collects signals from multiple sources, builds explicit reasoning chains, makes calibrated predictions with confidence intervals, and rigorously tracks its own accuracy using Brier scores. This is not gut-feel forecasting. Every prediction is backed by a documented reasoning chain. Every outcome is scored. The system learns from its mistakes.

​
Key Features

  • Multi-source signal collection: News, social, financial data, academic papers
  • Superforecasting methodology: Base rates, reference classes, Bayesian updating
  • Explicit reasoning chains: Base rate → evidence for/against → synthesis → confidence
  • Accuracy tracking: Brier score calculation, calibration analysis
  • Contrarian mode: Actively seeks counter-consensus predictions
  • Time horizons: 1 week to 1 year predictions

​
Activation

# Activate Predictor Hand
openfang hand activate predictor

# Activate for tech predictions
openfang hand activate predictor --settings "prediction_domain=tech,time_horizon=3_months"

# Enable contrarian mode
openfang hand activate predictor --settings "contrarian_mode=true"

​
Configuration Settings

​
prediction_domain
select
default:"tech"
Primary domain for predictions:
  • tech: Technology (default)
  • finance: Finance & markets
  • geopolitics: Geopolitics
  • climate: Climate & energy
  • general: Cross-domain
​
time_horizon
select
default:"3_months"
How far ahead to predict:
  • 1_week: 1 week
  • 1_month: 1 month
  • 3_months: 3 months (default)
  • 1_year: 1 year
​
data_sources
select
default:"all"
What types of sources to monitor for signals:
  • news: News only
  • social: Social media
  • financial: Financial data
  • academic: Academic papers
  • all: All sources (default)
​
report_frequency
select
default:"weekly"
How often to generate prediction reports:
  • daily: Daily
  • weekly: Weekly (default)
  • biweekly: Biweekly
  • monthly: Monthly
​
predictions_per_report
select
default:"5"
Number of predictions to include per report:
  • 3 predictions
  • 5 predictions (default)
  • 10 predictions
  • 20 predictions
​
track_accuracy
toggle
default:"true"
Score past predictions when their time horizon expires. Calculates Brier scores and calibration metrics.
​
confidence_threshold
select
default:"medium"
Minimum confidence to include a prediction:
  • low: 20%+ confidence
  • medium: 40%+ confidence (default)
  • high: 70%+ confidence
​
contrarian_mode
toggle
default:"false"
Actively seek and present counter-consensus predictions. For each consensus view, searches for evidence the consensus is wrong.

​
Required Tools

Predictor Hand requires access to these tools (all built-in):
  • shell_exec — Platform detection
  • file_read, file_write, file_list — Prediction ledger
  • web_fetch, web_search — Signal collection
  • memory_store, memory_recall — State and accuracy tracking
  • schedule_create, schedule_list, schedule_delete — Report scheduling
  • knowledge_add_entity, knowledge_add_relation, knowledge_query — Signal tracking

​
System Prompt Overview

Predictor Hand operates in 7 phases:
1

Platform Detection & State Recovery

Detects OS, loads previous predictions and accuracy data, reads configuration, queries knowledge graph for existing signals.
2

Schedule & Domain Setup

Creates report schedule. Builds domain-specific query templates (tech: product launches, funding; finance: earnings, macro indicators; etc.).
3

Signal Collection

Executes 20-40 targeted searches based on domain and data sources. Tags each signal by type (leading indicator, base rate, expert opinion, data point, anomaly), strength, direction, and credibility.
4

Accuracy Review

For predictions where resolution date has passed, searches for evidence, scores the prediction (correct/partially correct/incorrect/unresolvable), calculates Brier score, updates calibration metrics.
5

Pattern Analysis & Reasoning Chains

For each potential prediction, gathers relevant signals, builds reasoning chain (base rate → evidence for → evidence against → key uncertainties → reference class), applies cognitive bias checks.
6

Prediction Formulation

Structures each prediction: clear falsifiable claim, calibrated confidence (X%), time horizon, reasoning chain, key assumptions, resolution criteria. Filters by confidence threshold.
7

Report Generation

Generates prediction report with accuracy dashboard, active predictions, new predictions with reasoning, expired predictions with results. Saves to predictions_database.json. Updates dashboard statistics.

​
Usage Examples

​
Technology Predictions

openfang hand configure predictor \
  --set prediction_domain="tech" \
  --set time_horizon="3_months" \
  --set report_frequency="weekly"

openfang hand activate predictor
Example predictions:
  • “OpenAI will release GPT-5 before July 2025” (65% confidence)
  • “Meta will launch a competitive LLM within 6 months” (42% confidence)
  • “At least one major AI lab will face a significant safety incident” (28% confidence)

​
Financial Forecasting

openfang hand configure predictor \
  --set prediction_domain="finance" \
  --set data_sources="financial" \
  --set time_horizon="1_month"

openfang hand activate predictor
Example predictions:
  • “S&P 500 will close above 5000 by end of March” (58% confidence)
  • “Federal Reserve will cut rates by 0.25% in next meeting” (35% confidence)
  • “Bitcoin will trade above $70K within 30 days” (22% confidence)

​
Contrarian Predictions

openfang hand configure predictor \
  --set contrarian_mode="true" \
  --set confidence_threshold="low"

openfang hand activate predictor
Predictor Hand will identify consensus views and present evidence-based counter-narratives.

​
Prediction Structure

Each prediction follows this format: 
PREDICTION: OpenAI will release GPT-5 before July 1, 2025
CONFIDENCE: 65%
TIME HORIZON: July 1, 2025
DOMAIN: tech

REASONING CHAIN:
1. Base rate: GPT-4 was released 18 months after GPT-3. Historical cadence suggests 12-18 month cycles. Base rate: 50%

2. Key signals FOR (+15%):
   a. Leaked internal roadmap suggests Q2 2025 target (moderate signal, +8%)
   b. Compute scaling continues on track (OpenAI CEO statement, +4%)
   c. Hiring patterns show ML engineer surge in Q4 2024 (leading indicator, +3%)

3. Key signals AGAINST (-0%):
   [No strong contradicting signals found]

4. Net adjustment from base: +15%
   Final probability: 65%

KEY ASSUMPTIONS:
- No major safety incidents delay release
- Compute infrastructure remains available
- Competitive pressure from Anthropic/Google continues

RESOLUTION CRITERIA:
- Official blog post or product launch before July 1, 2025
- Model must be named "GPT-5" or confirmed as successor to GPT-4
- Source: OpenAI blog, TechCrunch, Verge

​
Accuracy Tracking

When track_accuracy is enabled, Predictor Hand scores all resolved predictions:
The gold standard for measuring prediction accuracy:
Brier Score = (predicted_probability - actual_outcome)^2

actual_outcome = 1 if prediction came true, 0 if not

Perfect score: 0.0 (always right with perfect confidence)
Coin flip: 0.25 (saying 50% on everything)
Terrible: 1.0 (100% confident, always wrong)

Good forecaster: < 0.15
Average forecaster: 0.20-0.30
Bad forecaster: > 0.35
Are your 70% predictions right ~70% of the time?
Predicted 90%+ → Actual: 85% (3 predictions)
Predicted 70-89% → Actual: 68% (12 predictions)
Predicted 50-69% → Actual: 52% (8 predictions)
Predicted 30-49% → Actual: 38% (5 predictions)
Predicted <30% → Actual: 15% (2 predictions)

Result: Well-calibrated

​
Dashboard Metrics

Predictor Hand tracks four key metrics:

Predictions Made

Total predictions ever made.

Accuracy

Overall accuracy percentage (resolved predictions only).

Reports Generated

Total prediction reports produced.

Active Predictions

Currently unresolved predictions.
View in the dashboard at http://localhost:4200/hands/predictor.

​
Report Format

# Prediction Report: tech
**Date**: 2025-03-07 | **Report #**: 12 | **Signals Analyzed**: 42

## Accuracy Dashboard
- Overall accuracy: 68% (25 predictions resolved)
- Brier score: 0.18 (good)
- Calibration: Well-calibrated

## Active Predictions
| # | Prediction | Confidence | Horizon | Status |
|---|-----------|------------|---------|--------|
| 1 | GPT-5 release before July 2025 | 65% | 2025-07-01 | Active |
| 2 | Meta launches competitive LLM | 42% | 2025-06-01 | Active |

## New Predictions This Report

### 1. OpenAI will release GPT-5 before July 1, 2025
[Full reasoning chain...]

### 2. Meta will launch a competitive LLM within 6 months
[Full reasoning chain...]

## Expired Predictions (Resolved This Cycle)

### Prediction #8: "Google will launch Gemini Ultra by March 2025"
- **Predicted**: 75% confidence
- **Outcome**: CORRECT — Launched Feb 15, 2025
- **Brier score**: 0.0625 (excellent)

## Signal Landscape
- Funding activity: Up 15% vs last month
- Major announcements: 3 significant product launches
- Sentiment: Bullish on AI infrastructure, bearish on consumer AI apps

## Meta-Analysis
Your accuracy on tech predictions is strongest for product launches (78%) and weakest for market adoption timelines (52%). Consider adjusting confidence downward for adoption-related predictions.

​
Superforecasting Principles

Predictor Hand applies these 10 principles:
  1. Triage: Focus on questions that are hard but not unknowable
  2. Break problems apart: Decompose into researchable sub-questions
  3. Balance inside and outside views: Use both specific evidence AND base rates
  4. Update incrementally: Adjust predictions as new evidence arrives
  5. Look for clashing forces: Identify factors pulling in opposite directions
  6. Distinguish signal from noise: Weight signals by reliability
  7. Calibrate: 70% predictions should come true ~70% of the time
  8. Post-mortem: Analyze why predictions went wrong
  9. Avoid the narrative trap: Compelling story ≠ likely outcome
  10. Aggregate views: Consider multiple perspectives

​
Best Practices

Predictor Hand will never express 0% or 100% confidence. Nothing is absolutely certain. Always maintains epistemic humility.
Start with confidence_threshold=medium. As the system builds track record, you can lower the threshold to capture more speculative predictions.
Enable contrarian_mode when you want to challenge consensus thinking. Useful for risk assessment and stress testing plans.
Review the Meta-Analysis section regularly. It identifies your forecasting strengths and weaknesses by domain and question type.

​
Advanced Configuration

​
Custom Prediction Requests

Manually trigger specific predictions: 
openfang chat predictor
> "Will Apple launch an AI-first device in 2025? Give me a calibrated forecast with reasoning."

​
Prediction Updates

When significant new evidence arrives: 
openfang chat predictor
> "Update prediction #12 based on today's OpenAI announcement."
Predictor Hand will revise the confidence and document the update in the ledger.

​
Cognitive Bias Checks

Before finalizing predictions, Predictor Hand checks for:
  1. Anchoring: Fixated on first number encountered
  2. Availability bias: Overweighting recent/memorable events
  3. Confirmation bias: Only looking for supporting evidence
  4. Narrative bias: Choosing prediction because it makes a good story
  5. Overconfidence: Too sure (if never wrong at this confidence level, probably overconfident)
  6. Scope insensitivity: Treating different scales the same
  7. Recency bias: Extrapolating recent trends too far
  8. Status quo bias: Defaulting to “nothing will change”

​
Next Steps

Collector Hand

Feed signals from Collector into Predictor for richer forecasts

Researcher Hand

Deep-dive research on prediction outcomes

Build docs developers (and LLMs) love

Get started for freeTalk to us