Overview
Metaculus is a forecasting platform that focuses on important questions about science, technology, and global events. Unlike traditional prediction markets, Metaculus uses a scoring system to reward accurate forecasters. The Metaculus API provides access to tournaments and forecasting questions.
MarketType Enum
from prediction_market_agent_tooling.markets.markets import MarketType
market_type = MarketType.METACULUS
Market Class
The MetaculusAgentMarket class extends AgentMarket and provides Metaculus-specific functionality.
from prediction_market_agent_tooling.markets.metaculus.metaculus import (
MetaculusAgentMarket,
)
Core Methods
Get Markets
Retrieve available forecasting questions from Metaculus.
from prediction_market_agent_tooling.markets.agent_market import FilterBy, SortBy
markets = MetaculusAgentMarket.get_markets(
limit=100,
tournament_id=32506,
filter_by=FilterBy.OPEN,
sort_by=SortBy.NEWEST
)
Maximum number of questions to retrieve
Specific tournament to fetch questions from
Filter questions by status (OPEN, RESOLVED, etc.)
Sort order (NEWEST, etc.)
markets
list[MetaculusAgentMarket]
List of forecasting questions matching the query criteria
Get Binary Market
Retrieve a specific binary question by ID.
market = MetaculusAgentMarket.get_binary_market(
id="question-id"
)
Question object containing details, resolution criteria, and metadata
Submit Prediction
Submit a forecast for a question.
# Binary question prediction
market.place_bet(
outcome="Yes",
amount=Probability(0.65)
)
Outcome being predicted (“Yes” or “No” for binary questions)
Probability estimate (0.0 to 1.0)
Metaculus uses probability estimates rather than monetary amounts. Predictions are scored based on accuracy.
Market Data Model
Question Properties
The forecasting question text
Detailed question description and background
Additional clarifications and edge cases
Explicit criteria for how the question will be resolved
Available outcomes (e.g., [“Yes”, “No”] for binary questions)
Community median prediction for “Yes” outcome
When the question closes for predictions
When the question was created
Direct link to the question on Metaculus
Whether the question is currently accepting predictions
Whether the current user has already predicted on this question
Final resolution outcome (if resolved)
Tournament Support
Metaculus organizes questions into tournaments with specific themes.
# AI Bot Tournament Q4
TOURNAMENT_ID_Q4 = 32506
markets = MetaculusAgentMarket.get_markets(
tournament_id=TOURNAMENT_ID_Q4,
filter_by=FilterBy.OPEN
)
Scoring System
Metaculus uses scoring mechanisms to evaluate forecast accuracy:
- Brier Score: Measures accuracy of probability estimates
- Peer Score: Performance relative to other forecasters
- Baseline Score: Performance vs. simple baselines
Repeat Predictions
Forecasters can update predictions as new information becomes available.
if not market.have_predicted or agent.repeat_predictions:
# Submit or update prediction
prediction = agent.answer_binary_market(market)
Question Structure
Metaculus questions include rich metadata:
full_question = f"""Question: {market.question}
Description: {market.description}
Fine Print: {market.fine_print}
Resolution Criteria: {market.resolution_criteria}"""
Real-World Examples
Tournament Agent
Dedicated agent for participating in Metaculus tournaments.
from prediction_market_agent.agents.metaculus_agent.deploy import (
DeployableMetaculusBotTournamentAgent,
)
class DeployableMetaculusBotTournamentAgent(DeployablePredictionAgent):
bet_on_n_markets_per_run = sys.maxsize # Predict on all available
dummy_prediction = False
repeat_predictions = False
tournament_id = 32506 # AI Bot Q4 tournament
supported_markets = [MarketType.METACULUS]
def get_markets(self, market_type: MarketType):
markets = MetaculusAgentMarket.get_markets(
limit=self.bet_on_n_markets_per_run,
tournament_id=self.tournament_id,
filter_by=FilterBy.OPEN,
sort_by=SortBy.NEWEST,
)
return markets
def verify_market(self, market_type: MarketType, market: AgentMarket) -> bool:
# Filter out if already predicted and not repeating
if not self.repeat_predictions and market.have_predicted:
return False
return True
Comprehensive Prediction
Use full question context for predictions.
def answer_binary_market(self, market: AgentMarket) -> ProbabilisticAnswer | None:
assert isinstance(market, MetaculusAgentMarket)
# Construct full question with all metadata
full_question = f"""Question: {market.question}
Question's description: {market.description}
Question's fine print: {market.fine_print}
Question's resolution criteria: {market.resolution_criteria}"""
# Make prediction using full context
prediction = self.agent.predict(full_question)
return (
prediction.outcome_prediction.to_probabilistic_answer()
if prediction.outcome_prediction is not None
else None
)
Integration with Prophet Agent
from prediction_market_agent.agents.prophet_agent.deploy import (
DeployablePredictionProphetGPTo1PreviewAgent,
)
class MetaculusTournamentAgent(DeployablePredictionAgent):
def load(self) -> None:
# Use high-performing model for Metaculus
self.agent = DeployablePredictionProphetGPTo1PreviewAgent(
enable_langfuse=self.enable_langfuse
)
def answer_binary_market(self, market: AgentMarket):
full_question = self.build_full_question(market)
prediction = self.agent.agent.predict(full_question)
return prediction.outcome_prediction.to_probabilistic_answer()
- Type: Forecasting platform (not a traditional prediction market)
- Currency: Points-based scoring system
- Website: https://www.metaculus.com
- Focus: Science, technology, global events, AI
- Community: Expert forecasters and researchers
- Resolution: Rigorous resolution process by Metaculus team
Tournament IDs
WARMUP_TOURNAMENT_ID = 3294
TOURNAMENT_ID_Q3 = 3349 # AI Bot Q3
TOURNAMENT_ID_Q4 = 32506 # AI Bot Q4
# AI Bot Tournament Q4
markets = MetaculusAgentMarket.get_markets(
tournament_id=TOURNAMENT_ID_Q4
)
Question Types
Metaculus supports various question types:
- Binary: Yes/No questions
- Numeric: Predicting a specific number
- Multiple Choice: Selecting from options
- Date: Predicting when something will occur
Prediction Strategy
Avoid Redundant Predictions
def verify_market(self, market_type: MarketType, market: AgentMarket) -> bool:
assert isinstance(market, MetaculusAgentMarket)
# Skip if already predicted (unless configured to repeat)
if not self.repeat_predictions and market.have_predicted:
return False
return True
Free Predictions
class MetaculusAgent(DeployablePredictionAgent):
# On Metaculus predictions are free
bet_on_n_markets_per_run = sys.maxsize
Advantages
- High-Quality Questions: Carefully curated forecasting questions
- Rich Context: Detailed descriptions and resolution criteria
- Expert Community: Skilled forecasters provide calibrated predictions
- No Cost: Free to participate and make predictions
- Educational: Learn from other forecasters
- Track Record: Build reputation through accurate forecasting
Considerations
Not a Prediction Market
Metaculus is a forecasting platform, not a traditional prediction market:
- No monetary trading
- Points-based scoring instead
- Focus on accuracy over profit
- Cannot “cash out” positions
Question Complexity
Metaculus questions often require deep analysis:
# Use complete question context
full_question = f"""Question: {market.question}
Description: {market.description}
Fine Print: {market.fine_print}
Resolution Criteria: {market.resolution_criteria}"""
Update Frequency
Allow repeat predictions to update forecasts:
repeat_predictions = True # Enable forecast updates
Error Handling
try:
market = MetaculusAgentMarket.get_binary_market(id=question_id)
if not market.have_predicted or self.repeat_predictions:
prediction = self.answer_binary_market(market)
if prediction:
market.place_bet("Yes", prediction.p_yes)
except Exception as e:
logger.error(f"Metaculus prediction failed: {e}")
Running Agents
# Run tournament agent
python prediction_market_agent/run_agent.py \
metaculus_bot_tournament_agent \
metaculus
Best Practices
- Use Full Context: Include description, fine print, and resolution criteria
- Check Predicted Status: Avoid duplicate predictions unless intended
- Tournament Focus: Participate in specific tournaments for consistency
- Model Selection: Use high-performing models for Metaculus complexity
- Regular Updates: Update predictions as new information emerges
See Also