Webhook Alert Integration

Overview

Webhook alerts allow you to integrate CronJob Guardian with any system that accepts HTTP requests. This enables:

Custom internal monitoring systems
Third-party incident management platforms
ITSM tools (ServiceNow, Jira Service Management)
Custom automation and remediation workflows
Data warehouses for alert analytics

Quick Start

Prepare your webhook endpoint

Ensure your endpoint:

Accepts POST requests (or configure a different method)
Returns 2xx status code on success
Can handle JSON payloads
Is accessible from your Kubernetes cluster

Create a Kubernetes Secret with the URL

kubectl create secret generic webhook-url \
  --namespace cronjob-guardian \
  --from-literal=url=https://your-system.example.com/api/alerts

If your endpoint requires authentication, add it to the secret:

kubectl create secret generic webhook-url \
  --namespace cronjob-guardian \
  --from-literal=url=https://your-system.example.com/api/alerts \
  --from-literal=token=your-api-token

Create a Webhook AlertChannel

kubectl apply -f - <<EOF
apiVersion: guardian.illenium.net/v1alpha1
kind: AlertChannel
metadata:
  name: custom-webhook
spec:
  type: webhook
  webhook:
    urlSecretRef:
      name: webhook-url
      namespace: cronjob-guardian
      key: url
    method: POST
    headers:
      Content-Type: application/json
EOF

Reference in CronJobMonitor

apiVersion: guardian.illenium.net/v1alpha1
kind: CronJobMonitor
metadata:
  name: monitored-jobs
  namespace: production
spec:
  selector:
    matchLabels:
      tier: critical
  alerting:
    channelRefs:
      - name: custom-webhook

Basic Webhook AlertChannel

Here’s the example from the repository:

alertchannels/webhook.yaml

# Generic Webhook AlertChannel
# Sends alerts to a custom HTTP endpoint
apiVersion: guardian.illenium.net/v1alpha1
kind: AlertChannel
metadata:
  name: custom-webhook
spec:
  type: webhook
  webhook:
    urlSecretRef:
      name: webhook-url
      namespace: cronjob-guardian
      key: url
    method: POST
    headers:
      Content-Type: application/json
      X-Custom-Header: guardian

Configuration Options

webhook.urlSecretRef

object

required

Reference to a Kubernetes Secret containing the webhook URL

Show properties

name

string

required

Name of the Secret

namespace

string

required

Namespace where the Secret exists

key

string

required

Key within the Secret (usually url)

webhook.method

string

HTTP method: POST, PUT, PATCH (default: POST)

webhook.headers

object

Custom HTTP headers to include in requestsCommon headers:

Content-Type: application/json
Authorization: Bearer <token> (use secrets for tokens)
Custom headers for your service

Webhook with Authentication

Most webhook endpoints require authentication. Here are common patterns:

apiVersion: guardian.illenium.net/v1alpha1
kind: AlertChannel
metadata:
  name: webhook-auth-bearer
spec:
  type: webhook
  webhook:
    urlSecretRef:
      name: webhook-credentials
      namespace: cronjob-guardian
      key: url
    method: POST
    headers:
      Content-Type: application/json
      Authorization: Bearer {\{.Secret.token}\}  # References secret key

Creating Secrets with Authentication

# Bearer token
kubectl create secret generic webhook-credentials \
  --namespace cronjob-guardian \
  --from-literal=url=https://api.example.com/alerts \
  --from-literal=token=your-bearer-token

# API key
kubectl create secret generic webhook-credentials \
  --namespace cronjob-guardian \
  --from-literal=url=https://api.example.com/alerts \
  --from-literal=api_key=your-api-key

# Basic auth (base64 encode "username:password")
kubectl create secret generic webhook-credentials \
  --namespace cronjob-guardian \
  --from-literal=url=https://api.example.com/alerts \
  --from-literal=basic_auth=$(echo -n "user:pass" | base64)

Webhook Payload Format

CronJob Guardian sends alerts as JSON payloads:

{
  "alertType": "jobFailed",
  "severity": "critical",
  "timestamp": "2026-03-04T08:15:23Z",
  "cronJob": {
    "name": "daily-backup",
    "namespace": "production",
    "uid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
  },
  "job": {
    "name": "daily-backup-28501234",
    "namespace": "production",
    "startTime": "2026-03-04T08:10:00Z",
    "completionTime": "2026-03-04T08:15:23Z",
    "duration": "5m23s",
    "status": "Failed",
    "exitCode": 1
  },
  "context": {
    "logs": "Error: Failed to connect to database\nConnection timeout after 30s\n...",
    "events": [
      {
        "type": "Warning",
        "reason": "BackoffLimitExceeded",
        "message": "Job has reached the specified backoff limit"
      }
    ],
    "podStatus": {
      "phase": "Failed",
      "containerStatuses": [
        {
          "name": "backup",
          "state": "terminated",
          "exitCode": 1,
          "reason": "Error"
        }
      ]
    },
    "suggestedFix": "Database connection failed. Check: 1. Database pod is running..."
  },
  "monitor": {
    "name": "database-backups",
    "namespace": "databases"
  },
  "dashboardURL": "https://guardian.example.com/jobs/production/daily-backup"
}

Alert Types

The alertType field can be:

jobFailed: Job execution failed
deadManTriggered: No successful run within expected time
slaBreached: Success rate fell below threshold
missedSchedule: Job didn’t start on schedule
durationRegression: Job duration increased significantly
suspendedTooLong: Job suspended for extended period

Integration Examples

ServiceNow Integration

apiVersion: guardian.illenium.net/v1alpha1
kind: AlertChannel
metadata:
  name: servicenow
spec:
  type: webhook
  webhook:
    urlSecretRef:
      name: servicenow-credentials
      namespace: cronjob-guardian
      key: url  # https://instance.service-now.com/api/now/table/incident
    method: POST
    headers:
      Content-Type: application/json
      Authorization: Basic {\{.Secret.auth}\}

Create the secret:

kubectl create secret generic servicenow-credentials \
  --namespace cronjob-guardian \
  --from-literal=url=https://yourinstance.service-now.com/api/now/table/incident \
  --from-literal=auth=$(echo -n "admin:password" | base64)

Microsoft Teams Integration

apiVersion: guardian.illenium.net/v1alpha1
kind: AlertChannel
metadata:
  name: teams
spec:
  type: webhook
  webhook:
    urlSecretRef:
      name: teams-webhook
      namespace: cronjob-guardian
      key: url  # https://outlook.office.com/webhook/...
    method: POST
    headers:
      Content-Type: application/json

Create the secret:

kubectl create secret generic teams-webhook \
  --namespace cronjob-guardian \
  --from-literal=url=https://outlook.office.com/webhook/your-webhook-url

Datadog Events API

apiVersion: guardian.illenium.net/v1alpha1
kind: AlertChannel
metadata:
  name: datadog
spec:
  type: webhook
  webhook:
    urlSecretRef:
      name: datadog-credentials
      namespace: cronjob-guardian
      key: url  # https://api.datadoghq.com/api/v1/events
    method: POST
    headers:
      Content-Type: application/json
      DD-API-KEY: {\{.Secret.api_key}\}

Create the secret:

kubectl create secret generic datadog-credentials \
  --namespace cronjob-guardian \
  --from-literal=url=https://api.datadoghq.com/api/v1/events \
  --from-literal=api_key=your-datadog-api-key

Custom Headers for Routing

Use custom headers to route alerts within your system:

apiVersion: guardian.illenium.net/v1alpha1
kind: AlertChannel
metadata:
  name: custom-routing
spec:
  type: webhook
  webhook:
    urlSecretRef:
      name: webhook-url
      namespace: cronjob-guardian
      key: url
    method: POST
    headers:
      Content-Type: application/json
      X-Alert-Source: cronjob-guardian
      X-Environment: production
      X-Team: platform
      X-Severity: "{\{.Alert.Severity}\}"  # Dynamic from alert

Your endpoint can route based on these headers:

X-Team: platform → Routes to platform team
X-Severity: critical → Escalates appropriately
X-Environment: production → Different handling than dev

Testing Webhook Integration

Set up a test endpoint

Use a service like webhook.site or requestbin.com to inspect payloads:

# Get a temporary webhook URL from webhook.site
# Then create the secret
kubectl create secret generic webhook-test \
  --namespace cronjob-guardian \
  --from-literal=url=https://webhook.site/your-unique-id

Create test AlertChannel

kubectl apply -f - <<EOF
apiVersion: guardian.illenium.net/v1alpha1
kind: AlertChannel
metadata:
  name: webhook-test
spec:
  type: webhook
  webhook:
    urlSecretRef:
      name: webhook-test
      namespace: cronjob-guardian
      key: url
    method: POST
    headers:
      Content-Type: application/json
EOF

Create a failing test job

kubectl apply -f - <<EOF
apiVersion: batch/v1
kind: CronJob
metadata:
  name: test-webhook
  namespace: default
  labels:
    test: webhook
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: test
              image: busybox
              command: ["sh", "-c", "echo 'Testing webhook' && exit 1"]
          restartPolicy: Never
EOF

And a monitor:

kubectl apply -f - <<EOF
apiVersion: guardian.illenium.net/v1alpha1
kind: CronJobMonitor
metadata:
  name: test-monitor
  namespace: default
spec:
  selector:
    matchLabels:
      test: webhook
  alerting:
    channelRefs:
      - name: webhook-test
EOF

Wait and inspect payload

After the job fails, check webhook.site to see the JSON payload sent by CronJob Guardian.Verify:

Headers are correct
Payload contains expected fields
Authentication works (if configured)

Clean up

kubectl delete cronjob test-webhook -n default
kubectl delete cronjobmonitor test-monitor -n default
kubectl delete alertchannel webhook-test
kubectl delete secret webhook-test -n cronjob-guardian

Troubleshooting

Webhook endpoint returns 4xx or 5xx

Check endpoint is accessible:

kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- \
  curl -X POST https://your-endpoint.com/api/alerts \
  -H "Content-Type: application/json" \
  -d '{"test": true}'

Check authentication:

Verify token/credentials in secret are correct
Ensure header format matches endpoint expectations
Check if credentials have expired

Check controller logs:

kubectl logs -n cronjob-guardian deployment/cronjob-guardian-controller-manager | grep -i webhook

Endpoint not receiving requests

Verify AlertChannel status:

kubectl describe alertchannel custom-webhook

Check network connectivity:

# Test from within the cluster
kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- \
  curl -v https://your-endpoint.com

Check for NetworkPolicies:

kubectl get networkpolicy -n cronjob-guardian

Wrong payload format

CronJob Guardian sends a standard JSON format. If your endpoint expects a different format:

Use a proxy/adapter service to transform the payload
Configure your endpoint to accept Guardian’s format
Use custom headers to provide routing/metadata

Example adapter pattern:

Guardian → Your Adapter Service → Third-party API
Adapter transforms payload to match third-party expectations

SSL/TLS certificate errors

If your endpoint uses self-signed certificates:

# Check certificate
openssl s_client -connect your-endpoint.com:443 -showcerts

Options:

Use a proper CA-signed certificate
Configure CronJob Guardian to trust your CA (requires controller configuration)
Use HTTP instead (not recommended for production)

Best Practices

Use Secrets for URLs

Always store webhook URLs in secrets, especially if they contain tokens or sensitive paths.

Implement Retry Logic

Your endpoint should be idempotent. Guardian may retry failed webhook calls.

Return 2xx Quickly

Respond with 200 OK quickly. Process alerts asynchronously if needed.

Validate Payloads

Verify payloads are from Guardian using custom headers or signature validation.

Log All Requests

Log incoming webhook requests for debugging and audit trails.

Monitor Endpoint Health

Alert if your webhook endpoint is down to avoid missing critical alerts.

Advanced: Webhook Signature Validation

For production, validate that webhooks are from CronJob Guardian:

apiVersion: guardian.illenium.net/v1alpha1
kind: AlertChannel
metadata:
  name: secure-webhook
spec:
  type: webhook
  webhook:
    urlSecretRef:
      name: webhook-credentials
      namespace: cronjob-guardian
      key: url
    method: POST
    headers:
      Content-Type: application/json
      X-Guardian-Secret: {\{.Secret.signing_key}\}

Your endpoint validates the X-Guardian-Secret header matches the expected value.

Next Steps

Email Alerts

Configure SMTP email notifications

Slack Alerts

Set up Slack integration

PagerDuty Alerts

Configure on-call escalation

Alert Channels Reference

Complete API documentation

Monitors

Alert Channels

Overview

Quick Start

Basic Webhook AlertChannel

Configuration Options

Webhook with Authentication

Creating Secrets with Authentication

Webhook Payload Format

Alert Types

Integration Examples

ServiceNow Integration

Microsoft Teams Integration

Datadog Events API

Custom Headers for Routing

Testing Webhook Integration

Troubleshooting

Best Practices

Use Secrets for URLs

Implement Retry Logic

Return 2xx Quickly

Validate Payloads

Log All Requests

Monitor Endpoint Health

Advanced: Webhook Signature Validation

Next Steps

Email Alerts

Slack Alerts

PagerDuty Alerts

Alert Channels Reference

Build docs developers (and LLMs) love

Monitors

Alert Channels

​Overview

​Quick Start

​Basic Webhook AlertChannel

​Configuration Options

​Webhook with Authentication

​Creating Secrets with Authentication

​Webhook Payload Format

​Alert Types

​Integration Examples

​ServiceNow Integration

​Microsoft Teams Integration

​Datadog Events API

​Custom Headers for Routing

​Testing Webhook Integration

​Troubleshooting

​Best Practices

Use Secrets for URLs

Implement Retry Logic

Return 2xx Quickly

Validate Payloads

Log All Requests

Monitor Endpoint Health

​Advanced: Webhook Signature Validation

​Next Steps

Email Alerts

Slack Alerts

PagerDuty Alerts

Alert Channels Reference

Build docs developers (and LLMs) love

Overview

Quick Start

Basic Webhook AlertChannel

Configuration Options

Webhook with Authentication

Creating Secrets with Authentication

Webhook Payload Format

Alert Types

Integration Examples

ServiceNow Integration

Microsoft Teams Integration

Datadog Events API

Custom Headers for Routing

Testing Webhook Integration

Troubleshooting

Best Practices

Advanced: Webhook Signature Validation

Next Steps