Overview
Every analysis generated by Argument Cartographer receives a “brutally honest” Credibility Score from 1-10. Unlike binary true/false ratings, this nuanced score reflects the complex reality of most debates.Philosophy: Even well-argued topics rarely score above 8/10. This intentional harshness helps users understand that most real-world debates have legitimate complexity and imperfect evidence.
Scoring Algorithm
The credibility score is calculated using a weighted multi-factor algorithm:Factor 1: Source Quality (30%)
Evaluates the diversity and reliability of sources used in the analysis.- Criteria
- Scoring
- Implementation
Positive factors:
- Number of independent sources (8+ is ideal)
- Domain diversity (different news outlets)
- Trusted outlet presence (Reuters, BBC, AP, etc.)
- Recency of sources (< 30 days)
- Geographic diversity (multiple regions)
- Echo chamber (all sources from same outlet)
- Outdated sources (> 1 year old)
- Unreliable domains (tabloids, conspiracy sites)
- Insufficient sourcing (< 3 sources)
Factor 2: Evidence Strength (30%)
Assesses the quality and type of evidence backing claims.- Evidence Hierarchy
- Scoring Logic
Strongest to Weakest:
- Primary Research (3 points)
- Peer-reviewed studies
- Original data/statistics
- Direct experiments
- Government reports with data
- Expert Testimony (2 points)
- Quotes from credentialed experts
- Academic analysis
- Professional organization statements
- Secondary Analysis (1 point)
- Journalism synthesizing research
- Think tank reports
- Meta-analyses
- Opinion/Speculation (0 points)
- Editorial opinions
- Anecdotal evidence
- Hypothetical scenarios
Factor 3: Fallacy Penalty (30%)
Deducts points based on detected logical fallacies.- Penalty Scale
- Implementation
| Fallacy Severity | Points Deducted | Max Penalty |
|---|---|---|
| Critical | -1.0 each | -3.0 total |
| Major | -0.5 each | -2.0 total |
| Minor | -0.2 each | -1.0 total |
- 90%+ confidence: Full penalty
- 70-89%: 75% of penalty
- 50-69%: 50% of penalty
- < 50%: 25% of penalty
Factor 4: Logical Coherence (10%)
Evaluates internal consistency and argument structure.- Criteria
- Scoring
Positive indicators:
- Clear thesis-claim-evidence links
- Both sides represented fairly
- Claims supported by evidence (not orphaned)
- Logical progression of arguments
- Orphaned claims (no evidence)
- One-sided analysis (no counterarguments)
- Circular reasoning in structure
- Disconnected evidence
Final Score Calculation
The algorithm is designed to be harsh - a score of 7-8 represents an excellent analysis. Scores of 9-10 are exceptionally rare and require near-perfect sourcing and zero fallacies.
Score Interpretation
- 9-10: Exceptional
- 7-8: Strong
- 5-6: Moderate
- 3-4: Weak
- 1-2: Very Weak
Characteristics:
- 8+ diverse, trusted sources
- Strong primary evidence throughout
- Zero or only minor fallacies
- Perfect logical structure
- Both sides thoroughly represented
UI Presentation
Credibility scores are displayed prominently:Score Breakdown Modal
Users can click the score to see detailed breakdown:Comparative Scoring
Users can compare scores across multiple analyses:Limitations & Transparency
What Scores DON’T Mean
Low Score ≠ False
Low Score ≠ False
A score of 3/10 means the available arguments are weak, not that the thesis is false. There might be:
- Limited public discourse on the topic
- Poor quality sources available
- Emerging issue without research yet
High Score ≠ True
High Score ≠ True
A score of 9/10 means the arguments are well-constructed, not that the thesis is proven. Scientific consensus can still evolve.
Score Comparison Isn't Direct
Score Comparison Isn't Direct
Comparing scores across topics has limits:
- Some topics have more research than others
- Controversial topics may have better sources (more coverage)
- Technical topics may lack accessible sources
Future Improvements
Planned enhancements to the scoring algorithm:Citation Quality
Weight sources based on impact factor, citations, and methodology rigor
Temporal Analysis
Track how scores change as new evidence emerges
Domain Expertise
Specialize scoring criteria for different fields (science, law, economics)
Crowdsourced Validation
Allow expert community to flag scoring issues
Next Steps
Fallacy Detection
Understand how fallacies impact scores
Source Quality
Learn about trusted source selection
Creating Analyses
Tips for generating high-credibility analyses
AI Orchestration
Technical details of score calculation
