Agent Configuration

Overview

The KloudMate Agent configuration controls agent behavior, authentication, remote configuration updates, and platform-specific settings. This configuration is separate from the OpenTelemetry Collector configuration to enable independent management and updates.

Configuration Structure

The agent configuration is defined by the Config struct in Go:

~/workspace/source/internal/config/config.go:14-26

type Config struct {
    // Collector configuration (OpenTelemetry collector config)
    Collector           map[string]interface{}
    AgentConfigPath     string
    OtelConfigPath      string
    ExporterEndpoint    string
    ConfigUpdateURL     string
    APIKey              string
    ConfigCheckInterval int
    DockerMode          bool
    DockerEndpoint      string
}

Configuration Parameters

Authentication

APIKey

string

required

Authentication key for KloudMate backend services.Sources:

CLI flag: --api-key
Environment: KM_API_KEY
Config file: key field

Example:

key: km_1234567890abcdef

Never commit API keys to version control. Use environment variables or secure secret management.

Endpoints

ExporterEndpoint

string

required

OpenTelemetry exporter endpoint URL for sending telemetry data.Sources:

CLI flag: --collector-endpoint
Environment: KM_COLLECTOR_ENDPOINT
Config file: endpoint field

Default: https://otel.kloudmate.com:4318Format: Must be a valid HTTP/HTTPS URL with optional portExamples:

# Production endpoint
endpoint: https://otel.kloudmate.com:4318

# Custom region
endpoint: https://otel.eu.kloudmate.com:4318

# On-premises deployment
endpoint: https://otel.company.internal:4318

# Development/testing
endpoint: http://localhost:4318

ConfigUpdateURL

string

API endpoint for fetching configuration updates.Sources:

CLI flag: --update-endpoint
Environment: KM_UPDATE_ENDPOINT
Config file: Not configurable via file

Default: Auto-derived from ExporterEndpointDerivation Logic: The update URL is automatically constructed from the exporter endpoint:

Extract root domain from exporter endpoint
Prepend api. subdomain
Append /agents/config-check path

// Example: https://otel.kloudmate.com:4318
// Becomes: https://api.kloudmate.com/agents/config-check

See implementation in ~/workspace/source/internal/config/config.go:28-52

Explicitly setting this value overrides the auto-derivation behavior.

Update Behavior

ConfigCheckInterval

integer

Interval in seconds between configuration update checks.Sources:

CLI flag: --config-check-interval
Environment: KM_CONFIG_CHECK_INTERVAL
Config file: interval field (duration format)

Default: 60 (seconds)Range: 10 to 3600 seconds recommendedExamples:

# Config file format (duration)
interval: 10s
interval: 5m
interval: 1h

# CLI/environment format (integer seconds)
--config-check-interval=60
export KM_CONFIG_CHECK_INTERVAL=300

Lower intervals enable faster configuration updates but increase API requests. For production, 60-300 seconds is recommended.

File Paths

AgentConfigPath

string

Path to the agent configuration YAML file.Sources:

CLI flag: --agent-config
Environment: KM_AGENT_CONFIG

Platform Defaults:

Linux/Unix: Not used (flags/env only)
macOS: Not used (flags/env only)
Windows: Not used (flags/env only)
Docker: Not used (flags/env only)

This parameter specifies where to find the agent config file for initial loading, not where to store runtime config.

OtelConfigPath

string

Path to the OpenTelemetry Collector configuration YAML file.Sources:

CLI flag: --config
Environment: KM_COLLECTOR_CONFIG

Platform Defaults:

Linux/Unix
macOS
Windows
Docker

/etc/kmagent/config.yaml

/Library/Application Support/kmagent/config.yaml

<executable-directory>\config.yaml

Where <executable-directory> is the directory containing kmagent.exe.

/etc/kmagent/config.yaml

Typically mounted from host or ConfigMap.

See implementation in ~/workspace/source/internal/config/config.go:54-70

Docker Mode

DockerMode

boolean

Enable Docker-specific configuration and monitoring.Sources:

CLI flag: --docker-mode
Environment: KM_DOCKER_MODE

Default: falseEffects:

Uses Docker-specific collector configuration template
Enables docker_stats receiver for container metrics
Monitors Docker socket for container events
Adjusts file paths for containerized environment

Example:

docker run -e KM_DOCKER_MODE=true \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  kloudmate/km-agent:latest

DockerEndpoint

string

Docker daemon socket endpoint for container monitoring.Sources:

CLI flag: --docker-endpoint
Environment: KM_DOCKER_ENDPOINT

Default: unix:///var/run/docker.sockFormats:

# Unix socket (default)
docker_endpoint: unix:///var/run/docker.sock

# TCP endpoint
docker_endpoint: tcp://localhost:2375

# TLS-enabled TCP
docker_endpoint: tcp://localhost:2376

Ensure the socket is mounted read-only in containers for security:

-v /var/run/docker.sock:/var/run/docker.sock:ro

Kubernetes Configuration

For Kubernetes deployments, additional configuration is available:

~/workspace/source/internal/config/config_k8s.go:18-37

type K8sAgentConfig struct {
    Logger    *zap.SugaredLogger
    K8sClient *kubernetes.Clientset
    StopCh    chan struct{}
    Version   string

    OtelCollectorConfig map[string]interface{}
    ExporterEndpoint    string
    ConfigUpdateURL     string
    APIKey              string
    ConfigCheckInterval string
    // Kubernetes specific
    KubeNamespace           string
    ClusterName             string
    DeploymentMode          string
    ConfigmapDaemonsetName  string
    ConfigmapDeploymentName string
    DaemonSetName           string
    DeploymentName          string
}

Kubernetes-Specific Parameters

KubeNamespace

string

Kubernetes namespace where the agent is deployed.Environment: KM_NAMESPACEDefault: km-agent

ClusterName

string

required

Unique identifier for the Kubernetes cluster.Environment: KM_CLUSTER_NAMEUsage: Added as k8s.cluster.name attribute to all telemetry data.

DeploymentMode

string

Kubernetes deployment mode for the agent.Values:

daemonset: Node-level monitoring (default)
deployment: Cluster-level monitoring
both: Both DaemonSet and Deployment

ConfigmapDaemonsetName

string

Name of ConfigMap containing DaemonSet collector configuration.Default: km-daemonset-config

ConfigmapDeploymentName

string

Name of ConfigMap containing Deployment collector configuration.Default: km-deployment-config

DaemonSetName

string

Name of the KloudMate agent DaemonSet.Default: km-agent-daemonset

DeploymentName

string

Name of the KloudMate agent Deployment.Default: km-agent-deployment

Environment Variables Reference

Required
Optional
Kubernetes

# Authentication
export KM_API_KEY="your-api-key"

# Exporter endpoint
export KM_COLLECTOR_ENDPOINT="https://otel.kloudmate.com:4318"

# Configuration paths
export KM_AGENT_CONFIG="/path/to/agent-config.yaml"
export KM_COLLECTOR_CONFIG="/path/to/collector-config.yaml"

# Update behavior
export KM_CONFIG_CHECK_INTERVAL="60"
export KM_UPDATE_ENDPOINT="https://api.kloudmate.com/agents/config-check"

# Docker mode
export KM_DOCKER_MODE="true"
export KM_DOCKER_ENDPOINT="unix:///var/run/docker.sock"

# Cluster identification
export KM_CLUSTER_NAME="production-cluster"

# Namespace
export KM_NAMESPACE="km-agent"

# Node identification (DaemonSet)
export KM_NODE_NAME="$(spec.nodeName)"

# ConfigMap names
export KM_DAEMONSET_CONFIG="km-daemonset-config"
export KM_DEPLOYMENT_CONFIG="km-deployment-config"

Configuration Examples

Minimal Configuration

agent-config.yaml

key: ${KM_API_KEY}
endpoint: https://otel.kloudmate.com:4318

Production Configuration

agent-config.yaml

# Authentication
key: ${KM_API_KEY}

# Endpoints
endpoint: https://otel.kloudmate.com:4318

# Update behavior
interval: 300s  # 5 minutes

# Debug output (development only)
debug: none

Docker Configuration

agent-config.yaml

key: ${KM_API_KEY}
endpoint: https://otel.kloudmate.com:4318
interval: 60s
debug: basic

Docker Compose

services:
  km-agent:
    image: kloudmate/km-agent:latest
    environment:
      - KM_API_KEY=${KM_API_KEY}
      - KM_COLLECTOR_ENDPOINT=https://otel.kloudmate.com:4318
      - KM_DOCKER_MODE=true
      - KM_CONFIG_CHECK_INTERVAL=60
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/log:/var/log:ro
      - ./agent-config.yaml:/etc/kmagent/agent-config.yaml:ro

Kubernetes Configuration

ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: km-agent-config
  namespace: km-agent
data:
  agent-config.yaml: |
    key: ${KM_API_KEY}
    endpoint: https://otel.kloudmate.com:4318
    interval: 120s

DaemonSet Environment

env:
  - name: KM_API_KEY
    valueFrom:
      secretKeyRef:
        name: km-agent-secret
        key: api-key
  - name: KM_COLLECTOR_ENDPOINT
    value: "https://otel.kloudmate.com:4318"
  - name: KM_CLUSTER_NAME
    value: "production-us-east-1"
  - name: KM_NODE_NAME
    valueFrom:
      fieldRef:
        fieldPath: spec.nodeName
  - name: KM_NAMESPACE
    valueFrom:
      fieldRef:
        fieldPath: metadata.namespace

Configuration Loading

The agent loads configuration through the LoadConfig() method:

~/workspace/source/internal/config/config.go:73-117

func (c *Config) LoadConfig() error {
    // Set environment variables for collector
    os.Setenv("KM_COLLECTOR_ENDPOINT", c.ExporterEndpoint)
    os.Setenv("KM_API_KEY", c.APIKey)

    // Auto-derive update URL if not set
    if c.ConfigUpdateURL == "" {
        c.ConfigUpdateURL = GetAgentConfigUpdaterURL(c.ExporterEndpoint)
    }

    // Determine config path based on platform
    configPath := c.OtelConfigPath
    if configPath == "" {
        if c.DockerMode {
            configPath = GetDockerConfigPath()
        } else {
            configPath = GetDefaultConfigPath()
        }
    }

    c.OtelConfigPath = configPath

    // Load config file if exists
    if _, err := os.Stat(configPath); err == nil {
        configData, err := os.ReadFile(configPath)
        if err != nil {
            return fmt.Errorf("failed to read config file: %w", err)
        }

        if err := yaml.Unmarshal(configData, c); err != nil {
            return fmt.Errorf("failed to parse config file: %w", err)
        }
    } else if !os.IsNotExist(err) {
        return fmt.Errorf("error checking config file: %w", err)
    }

    // Ensure config directory exists
    configDir := filepath.Dir(configPath)
    if err := os.MkdirAll(configDir, 0755); err != nil {
        return fmt.Errorf("failed to create config directory: %w", err)
    }

    return nil
}

Validation

The agent performs the following validation checks:

API Key Validation

Must be non-empty when connecting to authenticated endpoints
No format restrictions (supports various key formats)
Passed as Authorization header to all API requests

Endpoint Validation

Must be a valid URL with HTTP or HTTPS scheme
Port is optional (defaults to scheme default)
Hostname must be resolvable
Update URL is auto-derived if not explicitly set

Interval Validation

Must be a positive integer (seconds)
Recommended range: 10-3600 seconds
Too low values may cause API rate limiting

Path Validation

Configuration directory must be writable
Parent directories are created automatically
Collector config file must be valid YAML

Remote Configuration Updates

The agent periodically checks for configuration updates:

Send Status Update

POST request to ConfigUpdateURL with agent status:

{
  "is_docker": false,
  "hostname": "server-01",
  "platform": "linux",
  "architecture": "amd64",
  "agent_version": "0.1.0",
  "collector_version": "0.96.0",
  "agent_status": "running",
  "collector_status": "running",
  "last_error_message": ""
}

See implementation in ~/workspace/source/internal/updater/updater.go:64-121

Receive Update Response

API responds with update instructions:

{
  "restart_required": true,
  "config": {
    "receivers": {...},
    "processors": {...},
    "exporters": {...},
    "service": {...}
  }
}

Apply Configuration

If restart_required is true:

Write new configuration to temporary file
Atomically rename to actual config file
Restart OpenTelemetry Collector with new config

See implementation in ~/workspace/source/internal/updater/updater.go:123-159

Configuration updates are atomic operations. If an update fails, the previous configuration remains unchanged.

Troubleshooting

Configuration Not Loading

Symptom: Agent starts but doesn’t use configuration fileCauses:

File doesn’t exist at expected path
Invalid YAML syntax
Incorrect file permissions

Solution:

# Check file exists
ls -la /etc/kmagent/config.yaml

# Validate YAML syntax
yamllint /etc/kmagent/config.yaml

# Check permissions
chmod 644 /etc/kmagent/config.yaml

Environment Variables Not Expanded

Symptom: Literal ${VAR} appearing in configurationCauses:

Environment variable not set
Wrong syntax for variable expansion

Solution:

# Verify environment variables are set
env | grep KM_

# Use correct syntax
endpoint: ${KM_COLLECTOR_ENDPOINT}        # Correct
endpoint: $KM_COLLECTOR_ENDPOINT          # Wrong
endpoint: {KM_COLLECTOR_ENDPOINT}         # Wrong

Remote Updates Not Working

Symptom: Configuration never updates from serverCauses:

Update URL not reachable
Invalid API key
Firewall blocking outbound requests

Solution:

# Test connectivity
curl -X POST https://api.kloudmate.com/agents/config-check \
  -H "Authorization: ${KM_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"hostname":"test"}'

# Check agent logs
journalctl -u kmagent -f | grep -i config

CLI Commands

Configuration API

Overview

Configuration Structure

Configuration Parameters

Authentication

Endpoints

Update Behavior

File Paths

Docker Mode

Kubernetes Configuration

Kubernetes-Specific Parameters

Environment Variables Reference

Configuration Examples

Minimal Configuration

Production Configuration

Docker Configuration

Kubernetes Configuration

Configuration Loading

Validation

Remote Configuration Updates

Troubleshooting

Next Steps

Collector Configuration

Configuration Structure

Build docs developers (and LLMs) love

CLI Commands

Configuration API

​Overview

​Configuration Structure

​Configuration Parameters

​Authentication

​Endpoints

​Update Behavior

​File Paths

​Docker Mode

​Kubernetes Configuration

​Kubernetes-Specific Parameters

​Environment Variables Reference

​Configuration Examples

​Minimal Configuration

​Production Configuration

​Docker Configuration

​Kubernetes Configuration

​Configuration Loading

​Validation

​Remote Configuration Updates

​Troubleshooting

​Next Steps

Collector Configuration

Configuration Structure

Build docs developers (and LLMs) love

Overview

Configuration Structure

Configuration Parameters

Authentication

Endpoints

Update Behavior

File Paths

Docker Mode

Kubernetes Configuration

Kubernetes-Specific Parameters

Environment Variables Reference

Configuration Examples

Minimal Configuration

Production Configuration

Docker Configuration

Kubernetes Configuration

Configuration Loading

Validation

Remote Configuration Updates

Troubleshooting

Next Steps