Tarpit Responder

​

Overview

• Sends HTTP headers immediately (appears responsive)
• Streams content at a configurable slow rate (default: 24 bytes/second)
• Holds connections open for an extended period
• Wastes bandwidth, time, and resources of scrapers
• Can serve content from files, HTTP sources, or just hold the connection

​

Configuration

​

Required Parameters

​

Tarpit Configuration Block

​

Content Sources

​

File Content

tarpit_config {
    content file:///var/www/large-file.txt
    timeout 5m
    bytes_per_second 24
}

​

HTTP/HTTPS Content

tarpit_config {
    content https://www.example.com/robots.txt
    timeout 2m
    bytes_per_second 50
}

​

No Content (Timeout Only)

tarpit_config {
    timeout 1h
    bytes_per_second 24
}

​

Examples

localhost:8080 {
    defender tarpit {
        ranges openai aws deepseek
        tarpit_config {
            timeout 30s
            bytes_per_second 24
            response_code 200
        }
    }
    respond "Fast content for humans"
}

​

Implementation Details

​

Streaming Algorithm

1. Open content stream from configured source
2. Read first 512 bytes to detect content type
3. Send HTTP headers immediately (appears responsive)
4. Write first chunk to client
5. Start ticker that fires every 100ms
6. Each tick: Send bytes_per_second / 10 bytes
7. Continue until timeout or EOF reached
8. Close connection gracefully

chunk := make([]byte, r.Config.BytesPerSecond/10)

ticker := time.NewTicker(time.Millisecond * 100)
defer ticker.Stop()

timeout := time.After(r.Config.Timeout)

for {
    select {
    case <-ticker.C:
        // Send chunk every 100ms
    case <-timeout:
        // Stop after timeout
        return nil
    }
}

​

Content Type Detection

​

HTTP Response

​

Client Experience

​

What Scrapers See

$ time curl http://example.com
# Connection appears to work
# Headers received immediately
# Content trickles in extremely slowly...
# (waits 30 seconds for a few kilobytes)

real    0m30.524s
user    0m0.008s
sys     0m0.012s

​

Resource Impact on Scrapers

• Connection slots - Ties up connection pool
• Memory - Buffers accumulate slowly
• Time - Wastes significant wall-clock time
• Bandwidth - Over extended period, still uses bandwidth
• Processing - May trigger timeouts and retries

​

Use Cases

​

Maximum Scraper Annoyance

defender tarpit {
    ranges openai deepseek
    tarpit_config {
        content file:///var/www/100mb-junk.txt
        timeout 1h
        bytes_per_second 10
    }
}

​

AI Training Poisoning

defender tarpit {
    ranges openai
    tarpit_config {
        content file:///var/www/garbage-data.txt
        timeout 10m
        bytes_per_second 24
    }
}

​

Connection Exhaustion

defender tarpit {
    ranges scrapers
    tarpit_config {
        timeout 30m
        bytes_per_second 1
    }
}

​

Advantages

1. Resource Waste - Maximizes scraper resource consumption
2. Appears Valid - Returns 200 OK, scrapers think it’s working
3. Connection Exhaustion - Ties up connection pools
4. Time Waste - Scrapers spend ages getting minimal data
5. Configurable - Fine-tune annoyance level
6. No Blocking Signal - Scrapers can’t easily detect they’re being tarpitted

​

Disadvantages

1. Server Resources - Keeps connections open longer
2. Memory Usage - Each connection consumes server resources
3. Complexity - More complex than simple blocking
4. Bandwidth Over Time - Eventually sends the data (if content provided)

​

Comparison with Other Responders

• vs Block: Tarpit wastes resources, Block denies immediately
• vs Drop: Tarpit holds connection, Drop terminates it
• vs Garbage: Tarpit sends slowly, Garbage sends quickly
• vs Custom: Tarpit streams, Custom sends full response

​

When to Use Tarpit

• You want to waste maximum scraper resources
• Connection exhaustion is a goal
• Time-wasting is more valuable than bandwidth savings
• You want scrapers to think they’re succeeding (slowly)

• Server connection limits are tight
• You need to minimize resource usage
• Simple blocking is sufficient
• Legitimate users might be affected

​

Performance Considerations

​

Connection Limits

# Check active connections
netstat -an | grep ESTABLISHED | wc -l

# Check Caddy limits
caddy environ

​

Memory Usage

• HTTP buffers
• Content reader
• Ticker
• Response writer

top -p $(pgrep caddy)

​

Best Practices

1. Set reasonable timeouts - Don’t exhaust server resources (e.g., 30s-5m)
2. Monitor connection counts - Watch for resource exhaustion
3. Use small bytes_per_second - 10-50 for maximum annoyance
4. Serve garbage content - Combine with useless data
5. Target specific ranges - Don’t tarpit legitimate users
6. Test thoroughly - Ensure you’re not tarpitting yourself

​

Validation Errors

if r.Config.Timeout <= 0 {
    return errors.New("tarpit timeout must be greater than 0")
}
if r.Config.BytesPerSecond <= 10 {
    return errors.New("tarpit bytes_per_second must be greater than 10")
}

• tarpit timeout must be greater than 0
• tarpit bytes_per_second must be greater than 10
• unsupported tarpit Content protocol

​

Testing

# Time how long a request takes
time curl http://example.com

# Watch data trickle in
curl -v http://example.com

# Simulate blocked IP
time curl -H "X-Forwarded-For: 20.202.43.67" http://example.com

# Test with timeout
curl --max-time 5 http://example.com

​

Advanced Examples

​

Different Speeds for Different Sources

example.com {
    # Super slow for OpenAI
    defender tarpit {
        ranges openai
        tarpit_config {
            timeout 1h
            bytes_per_second 10
        }
    }
    
    # Moderately slow for AWS
    defender tarpit {
        ranges aws
        tarpit_config {
            timeout 5m
            bytes_per_second 100
        }
    }
    
    respond "Fast for humans"
}

​

Serve Realistic-Looking Content Slowly

api.example.com {
    defender tarpit {
        ranges scrapers
        tarpit_config {
            content file:///var/www/fake-api-response.json
            timeout 10m
            bytes_per_second 24
            response_code 200
            headers {
                Content-Type "application/json"
                X-API-Version "1.0"
            }
        }
    }
    
    reverse_proxy localhost:3000
}

​

Related Documentation

• Garbage Responder - Fast garbage delivery
• Drop Responder - Drop connections instead
• Block Responder - Simple blocking
• Network Tarpits - Background on tarpit technique