Skip to main content

​
Overview

Bot policy configuration defines rules for identifying and handling different types of traffic. Each rule specifies matching criteria and an action to take when traffic matches.

​
BotConfig

Defines a bot detection rule with matching criteria and action.

​
Type Definition

type BotConfig struct {
    UserAgentRegex *string           `json:"user_agent_regex,omitempty" yaml:"user_agent_regex,omitempty"`
    PathRegex      *string           `json:"path_regex,omitempty" yaml:"path_regex,omitempty"`
    HeadersRegex   map[string]string `json:"headers_regex,omitempty" yaml:"headers_regex,omitempty"`
    Expression     *ExpressionOrList `json:"expression,omitempty" yaml:"expression,omitempty"`
    Challenge      *ChallengeRules   `json:"challenge,omitempty" yaml:"challenge,omitempty"`
    Weight         *Weight           `json:"weight,omitempty" yaml:"weight,omitempty"`
    GeoIP          *GeoIP            `json:"geoip,omitempty"`
    ASNs           *ASNs             `json:"asns,omitempty"`
    Name           string            `json:"name" yaml:"name"`
    Action         Rule              `json:"action" yaml:"action"`
    RemoteAddr     []string          `json:"remote_addresses,omitempty" yaml:"remote_addresses,omitempty"`
}

​
Fields

​
Name
string
required
Unique identifier for this bot rule. Used in logs and metrics.
​
Action
Rule
required
Action to take when traffic matches this rule. See Rule enum.
​
UserAgentRegex
*string
Regular expression to match against the User-Agent header. Cannot be used with PathRegex.
​
PathRegex
*string
Regular expression to match against the request path. Cannot be used with UserAgentRegex.
​
HeadersRegex
map[string]string
Map of header names to regular expressions. All specified headers must match for the rule to apply.
​
Expression
*ExpressionOrList
CEL (Common Expression Language) expression for advanced matching logic. See ExpressionOrList.
​
RemoteAddr
[]string
List of CIDR ranges to match against the client IP address (e.g., "192.168.1.0/24").
​
Challenge
*ChallengeRules
Challenge configuration when Action is CHALLENGE. See challenge-rules.
​
Weight
*Weight
Weight adjustment when Action is WEIGH. If not set, defaults to {Adjust: 5}.
​
GeoIP
*GeoIP
Geographic IP matching configuration. See GeoIP.
​
ASNs
*ASNs
Autonomous System Number matching configuration. See ASNs.

​
Validation Rules

  • Must set Name
  • Must set at least one matching criterion: UserAgentRegex, PathRegex, HeadersRegex, RemoteAddr, Expression, GeoIP, or ASNs
  • Cannot set both UserAgentRegex and PathRegex
  • Regular expressions must not end with newline (use >- in YAML instead of >)
  • RemoteAddr entries must be valid CIDR notation
  • If Action is CHALLENGE, must set Challenge
  • If Action is WEIGH and Weight is nil, defaults to {Adjust: 5}

​
Example

bots:
  - name: Block malicious crawlers
    user_agent_regex: >(MJ12bot|AhrefsBot|SemrushBot)
    action: DENY

  - name: Challenge suspicious bots
    expression: weight > 10
    action: CHALLENGE
    challenge:
      algorithm: fast
      difficulty: 20

  - name: Weigh Russian IPs
    geoip:
      countries: ["ru"]
    action: WEIGH
    weight:
      adjust: 15

​
Rule Enum

Defines actions to take when traffic matches a bot rule.

​
Type Definition

type Rule string

const (
    RuleUnknown   Rule = ""
    RuleAllow     Rule = "ALLOW"
    RuleDeny      Rule = "DENY"
    RuleChallenge Rule = "CHALLENGE"
    RuleWeigh     Rule = "WEIGH"
    RuleBenchmark Rule = "DEBUG_BENCHMARK"
)

​
Values

​
ALLOW
string
Immediately allow the request without further checks.
​
DENY
string
Block the request with a configured status code.
​
CHALLENGE
string
Present a proof-of-work challenge to the client. Requires Challenge configuration.
​
WEIGH
string
Add weight to the client’s score. When thresholds are exceeded, triggers challenges or blocks.
​
DEBUG_BENCHMARK
string
Debug action for performance testing. Logs timing without affecting traffic.

​
ExpressionOrList

Flexible expression matching supporting single expressions or logical combinations.

​
Type Definition

type ExpressionOrList struct {
    Expression string   `json:"-" yaml:"-"`
    All        []string `json:"all,omitempty" yaml:"all,omitempty"`
    Any        []string `json:"any,omitempty" yaml:"any,omitempty"`
}

​
Fields

​
Expression
string
Single CEL expression. Can be set directly as a string in YAML/JSON.
​
All
[]string
List of CEL expressions that must all evaluate to true (logical AND).
​
Any
[]string
List of CEL expressions where at least one must evaluate to true (logical OR).

​
Validation Rules

  • Must set either Expression, All, or Any
  • Cannot set both All and Any

​
Example

# Single expression
expression: "weight > 10"

# All expressions must match (AND)
expression:
  all:
    - "weight > 5"
    - "request.headers['User-Agent'].contains('bot')"

# Any expression can match (OR)
expression:
  any:
    - "geoip.country == 'CN'"
    - "geoip.country == 'RU'"

​
GeoIP

Geographic IP address matching based on country codes.

​
Type Definition

type GeoIP struct {
    Countries []string `json:"countries"`
}

​
Fields

​
Countries
[]string
required
List of two-letter ISO 3166-1 alpha-2 country codes (case-insensitive).

​
Validation Rules

  • Country codes must be exactly 2 letters matching [a-zA-Z]{2}
  • Country codes are normalized to lowercase

​
Example

geoip:
  countries: ["cn", "ru", "kp"]

​
ASNs

Autonomous System Number (ASN) matching for network-based rules.

​
Type Definition

type ASNs struct {
    Match []uint32 `json:"match"`
}

​
Fields

​
Match
[]uint32
required
List of ASNs to match against the client’s network.

​
Validation Rules

  • ASNs must not be in private use ranges (RFC 6996):
    • 64512-65534
    • 4200000000-4294967294

​
Example

asns:
  match: [15169, 16509]  # Google and Amazon

​
Weight

Score adjustment for traffic matching a WEIGH rule.

​
Type Definition

type Weight struct {
    Adjust int `json:"adjust" yaml:"adjust"`
}

​
Fields

​
Adjust
int
required
Integer value to add to the client’s weight score. Can be negative to reduce weight.

​
Example

weight:
  adjust: 10

​
ImportStatement

Import bot rules from external files.

​
Type Definition

type ImportStatement struct {
    Import string      `json:"import"`
    Bots   []BotConfig
}

​
Fields

​
Import
string
required
Path to YAML file containing bot rules. Use (data)/ prefix to load from embedded data.

​
Example

bots:
  - import: "(data)/botPolicies.yaml"
  - name: Custom rule
    user_agent_regex: "CustomBot"
    action: DENY

​
Errors

var (
    ErrNoBotRulesDefined                 = errors.New("config: must define at least one (1) bot rule")
    ErrBotMustHaveName                   = errors.New("config.Bot: must set name")
    ErrBotMustHaveUserAgentOrPath        = errors.New("config.Bot: must set either user_agent_regex, path_regex, headers_regex, or remote_addresses")
    ErrBotMustHaveUserAgentOrPathNotBoth = errors.New("config.Bot: must set either user_agent_regex, path_regex, and not both")
    ErrUnknownAction                     = errors.New("config.Bot: unknown action")
    ErrInvalidUserAgentRegex             = errors.New("config.Bot: invalid user agent regex")
    ErrInvalidPathRegex                  = errors.New("config.Bot: invalid path regex")
    ErrInvalidHeadersRegex               = errors.New("config.Bot: invalid headers regex")
    ErrInvalidCIDR                       = errors.New("config.Bot: invalid CIDR")
    ErrRegexEndsWithNewline              = errors.New("config.Bot: regular expression ends with newline (try >- instead of > in yaml)")
    ErrNotCountryCode                    = errors.New("config.Bot: invalid country code")
    ErrPrivateASN                        = errors.New("bot.ASNs: you have specified a private use ASN")
)

Build docs developers (and LLMs) love

Get started for freeTalk to us