Overview
Auto-instrumentation allows you to collect traces, metrics, and logs from your applications without modifying application code or rebuilding container images. The agent injects OpenTelemetry instrumentation libraries at runtime using init containers and environment variables.How It Works
The auto-instrumentation system operates through three key components:- OpenTelemetry Operator: Watches for annotated pods and injects instrumentation
- Instrumentation CRD: Defines language-specific instrumentation configuration
- Agent Annotation Helper: Generates proper annotations for deployment rollouts
Auto-instrumentation is currently available only for Kubernetes deployments. Linux, Docker, and Windows installations require manual instrumentation.
Supported Languages
The KloudMate Agent supports auto-instrumentation for the following languages:Node.js
Automatically instruments Node.js applications using the official OpenTelemetry Node.js SDK
Java
Supports Java applications with the OpenTelemetry Java agent
Python
Instruments Python applications using OpenTelemetry Python SDK
.NET
Auto-instruments .NET applications with OpenTelemetry .NET SDK
Go
Supports Go applications (requires eBPF-based instrumentation)
Installation Prerequisites
Before enabling auto-instrumentation, ensure you have:Install KloudMate Agent with APM Enabled
Install the agent with the
featuresEnabled.apm flag set to true:Enabling Auto-Instrumentation
To instrument your application, add the appropriate annotation to your deployment’s pod template:Language-Specific Annotations
Annotation Format
The annotation follows this pattern:<language>: One ofnodejs,java,python,dotnet, orgo<namespace>: The namespace where the Instrumentation CRD is installed (default:km-agent)<instrumentation-crd-name>: Name of the Instrumentation resource (default:km-agent-instrumentation-crd)
Programmatic Annotation Generation
The agent provides a helper function to generate instrumentation annotations programmatically:- Accepts a language identifier and enabled flag
- Reads namespace and CRD name from environment variables
- Generates proper Kubernetes annotations for deployment patching
- Includes a
restartedAtannotation to trigger automatic pod rollout
Configuration
Helm Chart Configuration
The auto-instrumentation feature is configured through Helm values:values.yaml:14-18,249-261
Instrumentation CRD Resource
The Instrumentation CRD defines configuration for all supported languages:Example Instrumentation Resource
Verification
After adding instrumentation annotations:Check Pod Injection
Verify that the OpenTelemetry init container was injected:Look for an init container named
opentelemetry-auto-instrumentationCheck Environment Variables
Verify OpenTelemetry environment variables were set:Expected output includes:
Troubleshooting
Pods not being instrumented
Pods not being instrumented
Symptoms: Annotations are present but no init container is injectedSolutions:
- Verify the OpenTelemetry Operator is running:
- Check operator logs:
- Ensure the Instrumentation CRD exists:
Application crashes after instrumentation
Application crashes after instrumentation
Symptoms: Pods fail to start after adding instrumentationSolutions:
- Check pod logs for initialization errors
- Verify the instrumentation image is compatible with your application version
- Check resource limits - instrumentation may require additional memory
No telemetry data received
No telemetry data received
Symptoms: Instrumentation appears successful but no data in dashboardSolutions:
- Verify the exporter endpoint is correct:
- Check network connectivity from pods to the collector endpoint
- Verify API key is correctly configured
Wrong language detected
Wrong language detected
Symptoms: The operator injects the wrong instrumentation librarySolutions:
- Ensure you’re using the correct annotation for your language
- Check for conflicting annotations on the pod
- Verify the language identifier in the annotation matches exactly (case-sensitive)
Best Practices
Start with Staging
Test auto-instrumentation in non-production environments first to validate overhead and compatibility
Monitor Resource Usage
Auto-instrumentation adds overhead. Monitor CPU and memory usage after enabling instrumentation
Use Sampling
Configure sampling rates to reduce data volume in high-traffic applications
Version Compatibility
Keep instrumentation images updated to ensure compatibility with the latest application frameworks
Next Steps
Lifecycle Management
Learn how the agent manages collector lifecycle and configuration updates
Multi-Platform Support
Explore agent deployment across Linux, Docker, Windows, and Kubernetes