Skip to main content

​
Supported AI Services

Caddy Defender includes IP ranges for major AI services and their web crawlers. These are commonly used to block or allow AI bots that index content for training data.

​
OpenAI

​
openai
string
IP ranges for OpenAI services including ChatGPT, GPTBot, and SearchBot.Sources:Fetcher: fetchers.OpenAIFetcher in ranges/fetchers/openai.go:19-71Documentation: OpenAI Crawler Overview

​
OpenAI Data Structure

The fetcher aggregates ranges from three separate endpoints: 
type OpenAIIPRanges struct {
    CreationTime string `json:"creationTime"`
    Prefixes     []struct {
        IPv4Prefix string `json:"ipv4Prefix"`
    } `json:"prefixes"`
}

​
Example Usage

# Block OpenAI crawlers
defender block {
    ranges openai
}

# Block all AI services
defender block {
    ranges openai deepseek githubcopilot mistral
}

​
DeepSeek

​
deepseek
string
Hardcoded IP ranges for DeepSeek AI services.Source: DeepSource IP DocumentationFetcher: fetchers.DeepSeekFetcher in ranges/fetchers/deepseek.go:12-20Default Ranges:
  • 35.225.112.198/32
  • 34.42.70.44/32
  • 104.154.172.152/32
DeepSeek ranges are hardcoded as the service provides a static list rather than a dynamic API endpoint.

​
Example Usage

defender block {
    ranges deepseek
}

​
GitHub Copilot

​
githubcopilot
string
IP ranges for GitHub Copilot services.Source: GitHub Meta APIFetcher: fetchers.GithubCopilotFetcher in ranges/fetchers/github.go:22-68Documentation: GitHub IP Addresses

​
Fetcher Implementation

The GitHub Copilot fetcher extracts the copilot key from the GitHub Meta API: 
func (f GithubCopilotFetcher) FetchIPRanges() ([]string, error) {
    resp, err := http.Get("https://api.github.com/meta")
    // ... error handling
    
    var result map[string]interface{}
    json.Unmarshal(body, &result)
    
    copilotValue := result["copilot"]
    // Convert to []string
}

​
Example Usage

defender block {
    ranges githubcopilot
}

​
Mistral AI

​
mistral
string
IP ranges for Mistral AI services.Source: Mistral AI IPs JSONFetcher: fetchers.MistralFetcher in ranges/fetchers/mistral.go:21-53

​
Mistral Data Structure

var ipRanges struct {
    Prefixes []struct {
        IPv4Prefix string `json:"ipv4Prefix"`
    } `json:"prefixes"`
}

​
Example Usage

defender block {
    ranges mistral
}

​
Common Use Cases

Prevent AI services from indexing your content:
defender block {
    ranges openai deepseek githubcopilot mistral
    message "AI crawlers are not permitted on this site"
}
Block all AI services except GitHub Copilot:
defender block {
    ranges openai deepseek mistral
}
Instead of blocking, apply rate limiting:
defender ratelimit {
    ranges openai deepseek githubcopilot mistral
}
Return a robots.txt file that disallows crawling:
defender custom {
    ranges openai deepseek githubcopilot mistral
    serve_ignore
}
This will serve a robots.txt that disallows all paths when AI bots request it.

​
Combining with Custom Ranges

You can mix AI service ranges with custom CIDR blocks: 
defender block {
    # Block OpenAI and some custom IPs
    ranges openai 203.0.113.0/24 198.51.100.0/24
}
Before blocking AI crawlers, consider:
  • Your robots.txt policy
  • Terms of service for AI companies
  • Potential impact on discoverability
  • Legal jurisdictions regarding data scraping

​
Monitoring AI Traffic

To monitor AI bot traffic before blocking:
  1. Use the custom responder to log requests: 
    defender custom {
    ranges openai deepseek githubcopilot mistral
    message "AI bot detected"
    status_code 200
}
  2. Analyze your logs for AI bot patterns
  3. Adjust your blocking strategy based on findings

​
Keeping Ranges Updated

AI services may update their IP ranges periodically:
  1. Rebuild regularly: Run go run ranges/main.go to fetch latest ranges
  2. Monitor announcements: Follow AI service blogs for IP range updates
  3. Automate updates: Include range updates in your deployment pipeline
Hardcoded ranges (like DeepSeek) may require manual updates if the service announces changes.

Build docs developers (and LLMs) love

Get started for freeTalk to us