Custom Fetcher

Overview

Creating a custom fetcher allows you to integrate IP ranges from any source into Caddy Defender. This guide walks through implementing the IPRangeFetcher interface and integrating your fetcher into the build system.

Creating a Custom Fetcher

Create the fetcher file

Create a new .go file in the ranges/fetchers/ directory. The filename should match your service name (e.g., myservice.go).

cd ranges/fetchers/
touch myservice.go

Implement the IPRangeFetcher interface

Your fetcher must implement all three methods of the IPRangeFetcher interface defined in ranges/fetchers/fetcher.go:4:

package fetchers

// MyServiceFetcher implements the IPRangeFetcher interface for My Service.
type MyServiceFetcher struct{}

func (f MyServiceFetcher) Name() string {
    return "MyService"
}

func (f MyServiceFetcher) Description() string {
    return "Fetches IP ranges for My Service."
}

func (f MyServiceFetcher) FetchIPRanges() ([]string, error) {
    return []string{
        "203.0.113.0/24",
        "2001:db8::/32",
    }, nil
}

The Name() method’s return value will be converted to lowercase and used as the key in the generated IP ranges map. Choose a descriptive, unique name.

Add your fetcher to the fetchersList

Open ranges/main.go and add your fetcher to the fetchersList array around line 34:

ranges/main.go

fetchersList := []fetchers.IPRangeFetcher{
    fetchers.VPNFetcher{},
    fetchers.LinodeFetcher{},
    // ... other fetchers ...
    fetchers.MyServiceFetcher{}, // Add your fetcher here
}

For configurable fetchers, you can instantiate them with specific parameters:

ranges/main.go

fetchersList := []fetchers.IPRangeFetcher{
    // ... other fetchers ...
    fetchers.MyServiceFetcher{},                    // Global ranges
    fetchers.MyServiceFetcher{Region: "us-east-1"}, // Region-specific
    fetchers.MyServiceFetcher{Region: "eu-west-1"}, // Another region
}

Regenerate the generated.go file

Run the ranges generator to fetch IP ranges and create the updated ranges/data/generated.go file:

cd ranges/
go run main.go

You should see output like:

🚀 Starting MyService: Fetches IP ranges for My Service.
✅ Completed MyService: Fetched 42 IP ranges
🎉 All 1337 IP ranges have been successfully written to ranges/data/generated.go

The generator runs all fetchers concurrently, so the order of output may vary. If a fetcher fails, you’ll see an error message with details.

Verify the output

Check that your IP ranges were added to ranges/data/generated.go:

ranges/data/generated.go

var IPRanges = map[string][]string{
    // ... other services ...
    "myservice": {
        "203.0.113.0/24",
        "2001:db8::/32",
    },
}

The key will be the lowercase version of your Name() return value.

Test your fetcher

You can test your fetcher in isolation by creating a simple test program:

test_fetcher.go

package main

import (
    "fmt"
    "pkg.jsn.cam/caddy-defender/ranges/fetchers"
)

func main() {
    f := fetchers.MyServiceFetcher{}
    ranges, err := f.FetchIPRanges()
    if err != nil {
        fmt.Printf("Error: %v\n", err)
        return
    }
    fmt.Printf("Fetched %d ranges:\n", len(ranges))
    for _, r := range ranges {
        fmt.Printf("  %s\n", r)
    }
}

Run it with:

go run test_fetcher.go

Interface Requirements

Your fetcher must satisfy these requirements:

Name()

string

required

Must return: A unique, descriptive name for your serviceUsed for: Map key in generated output (converted to lowercase)Example: "MyService" becomes key "myservice"

Description()

string

required

Must return: A human-readable description of what the fetcher doesUsed for: Progress logging during executionExample: "Fetches IP ranges for My Service API"

FetchIPRanges()

([]string, error)

required

Must return:

A slice of CIDR-formatted IP ranges (IPv4 and/or IPv6)
An error if the fetch operation fails

CIDR format examples:

IPv4: "192.0.2.0/24", "203.0.113.128/25"
IPv6: "2001:db8::/32", "2001:db8:1234::/48"

Error handling: Return descriptive errors that help diagnose issues:

return nil, fmt.Errorf("failed to fetch from %s: %v", url, err)

Best Practices

Error Handling

Always return descriptive errors that include:

What operation failed
The URL or source being accessed
The underlying error

if resp.StatusCode != http.StatusOK {
    return nil, fmt.Errorf("received non-200 status code from MyService: %d", resp.StatusCode)
}

if err := json.Unmarshal(body, &result); err != nil {
    return nil, fmt.Errorf("failed to unmarshal MyService JSON: %v", err)
}

HTTP Requests

Always close response bodies: defer resp.Body.Close()
Check HTTP status codes before processing
Set reasonable timeouts for HTTP clients
Handle rate limiting if applicable

client := &http.Client{
    Timeout: 30 * time.Second,
}
resp, err := client.Get(url)
if err != nil {
    return nil, fmt.Errorf("request failed: %v", err)
}
defer resp.Body.Close()

Data Validation

Validate that fetched data is in the correct CIDR format:

import "net"

func validateCIDR(cidr string) error {
    _, _, err := net.ParseCIDR(cidr)
    return err
}

Filter out invalid entries:

validRanges := []string{}
for _, r := range rawRanges {
    if validateCIDR(r) == nil {
        validRanges = append(validRanges, r)
    }
}
return validRanges, nil

Performance

Pre-allocate slices when the size is known:

ipRanges := make([]string, 0, expectedSize)

Use streaming JSON decoding for large responses:
```
json.NewDecoder(resp.Body).Decode(&result)
```
Consider caching if the API has rate limits

Naming Conventions

Struct name: MyServiceFetcher
File name: myservice.go
Package: fetchers
Name() return: "MyService" (will become "myservice" in output)

Real-World Examples

Refer to these existing fetchers for implementation patterns:

Simple Static Ranges

See ranges/fetchers/private.go:12 - Returns hardcoded RFC 1918 private IP ranges

JSON API with Struct

See ranges/fetchers/cloudflare.go:20 - Fetches from Cloudflare’s API and parses JSON response

Multiple API Endpoints

See ranges/fetchers/openai.go:19 - Combines results from multiple OpenAI JSON endpoints

Text File Parsing

See ranges/fetchers/vpn.go:21 - Fetches and parses newline-delimited IP ranges from a text file

Configurable with Validation

See ranges/fetchers/asn.go:24 - Accepts ASN list parameter with validation in constructor

Subdirectory Organization

See ranges/fetchers/aws/ - Complex fetchers can be organized in subdirectories with shared utilities

Troubleshooting

Fetcher not executing

Problem: Your fetcher doesn’t appear in the outputSolutions:

Verify you added it to fetchersList in ranges/main.go:34
Check for compilation errors: go build ./ranges
Ensure the package name is fetchers

Error: oldString not found

Problem: Fetch completes but ranges aren’t in outputSolutions:

Check that FetchIPRanges() returns non-empty slice
Verify CIDR format is correct
Look for errors in console output (❌ messages)

Import cycle errors

Problem: import cycle not allowedSolutions:

Don’t import ranges/main from your fetcher
Use the fetchers package, not a subdirectory
For complex fetchers, create a subdirectory like fetchers/myservice/

Timeout or slow fetching

Problem: Fetcher takes too long or times outSolutions:

Add HTTP client timeout:

client := &http.Client{Timeout: 30 * time.Second}

Check if the API requires authentication
Verify the URL is accessible from your network
Consider pagination for large datasets

Advanced: Conditional Fetchers

Some fetchers are only added to the list based on flags or conditions:

Flag-Based Inclusion

From ranges/main.go:57:

if fetchTor {
    fetchersList = append(fetchersList, fetchers.TorFetcher{})
}

Dynamic Fetcher Creation

From ranges/main.go:64:

if asnList != "" {
    asns := strings.Split(asnList, ",")
    fetchersList = append(fetchersList, fetchers.NewASNFetcher(asns))
}

You can add your own flags in ranges/main.go:

var (
    outputFormat string
    outputFile   string
    myServiceKey string // Add your flag
)

func main() {
    flag.StringVar(&myServiceKey, "myservice-key", "", "API key for MyService")
    flag.Parse()
    
    // ...
    
    if myServiceKey != "" {
        fetchersList = append(fetchersList, fetchers.MyServiceFetcher{
            APIKey: myServiceKey,
        })
    }
}

Responders

IP Ranges

Fetchers

Overview

Creating a Custom Fetcher

Interface Requirements

Best Practices

Real-World Examples

Simple Static Ranges

JSON API with Struct

Multiple API Endpoints

Text File Parsing

Configurable with Validation

Subdirectory Organization

Troubleshooting

Advanced: Conditional Fetchers

Flag-Based Inclusion

Dynamic Fetcher Creation

Next Steps

Fetchers Overview

Using IP Ranges

Build docs developers (and LLMs) love

Responders

IP Ranges

Fetchers

​Overview

​Creating a Custom Fetcher

​Interface Requirements

​Best Practices

​Real-World Examples

​Simple Static Ranges

​JSON API with Struct

​Multiple API Endpoints

​Text File Parsing

​Configurable with Validation

​Subdirectory Organization

​Troubleshooting

​Advanced: Conditional Fetchers

​Flag-Based Inclusion

​Dynamic Fetcher Creation

​Next Steps

Fetchers Overview

Using IP Ranges

Build docs developers (and LLMs) love

Overview

Creating a Custom Fetcher

Interface Requirements

Best Practices

Real-World Examples

Simple Static Ranges

JSON API with Struct

Multiple API Endpoints

Text File Parsing

Configurable with Validation

Subdirectory Organization

Troubleshooting

Advanced: Conditional Fetchers

Flag-Based Inclusion

Dynamic Fetcher Creation

Next Steps