Debugging Terraform

This guide covers debugging techniques for Terraform, from automated tests to complex runtime operations using real configurations.

Overview

Terraform is written in Go, which means you can use standard Go debugging tools:

Delve - Go debugger
GoLand - JetBrains IDE with built-in debugging
VS Code Go Extension - VS Code debugging support

Debugging Automated Tests

The most straightforward debugging workflow is debugging automated tests.

VS Code Test Debugging

The Go extension adds run test | debug test options above all tests in *_test.go files:

Open a test file (e.g., internal/command/test_test.go)
Look for the run test | debug test options above each test function
Click debug test to start debugging immediately

No prior configuration required!

Custom Test Launch Configuration

For more control (e.g., environment variables), create a launch configuration:

// .vscode/launch.json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Run selected test",
      "type": "go",
      "request": "launch",
      "mode": "test",
      "program": "${workspaceFolder}",
      "args": [
        "-test.run",
        "^${selectedText}$"
      ],
      "env": {
        "TF_LOG": "DEBUG"
      }
    }
  ]
}

Usage:

Highlight a test name in the editor
Start the debugger with “Run selected test” configuration

Source: docs/debugging.md:22-29

Debugging Terraform Operations

Method 1: Delve CLI Tool

This approach gives you maximum control over the debugging process.

Step 1: Compile with Debug Flags

go install -gcflags="all=-N -l"

This disables optimizations and inlining, making debugging easier.

Step 2: Start Debug Server

# Navigate to your Terraform configuration directory
cd /path/to/terraform/config

# Start debug server (adjust terraform binary path if needed)
dlv exec $HOME/go/bin/terraform --headless --listen :2345 --log -- apply

Replace apply with any Terraform command you want to debug. Source: docs/debugging.md:42-55

Step 3a: Connect via Delve CLI

dlv connect :2345

Now you can use Delve commands:

# Set breakpoint
(dlv) break main.main

# Continue execution
(dlv) continue

# Step through code
(dlv) step

# Print variables
(dlv) print variableName

Source: docs/debugging.md:57-63

Step 3b: Connect via VS Code

Create a launch configuration:

// .vscode/launch.json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Connect to dlv server",
      "type": "go",
      "request": "attach",
      "mode": "remote",
      "remotePath": "${workspaceFolder}",
      "port": 2345,
      "host": "127.0.0.1"
    }
  ]
}

Then select “Connect to dlv server” from the debug panel. Source: docs/debugging.md:65-72

Method 2: VS Code Debugger (Direct Launch)

Launch Terraform directly from VS Code without starting a separate debug server.

Step 1: Create Launch Configuration

// .vscode/launch.json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Run Terraform in debug mode",
      "type": "go",
      "request": "launch",
      "mode": "debug",
      "program": "${workspaceFolder}",
      "args": [
        "apply",
        "-auto-approve"
      ],
      "cwd": "/path/to/terraform/config",
      "env": {
        "TF_LOG": "DEBUG"
      }
    }
  ]
}

Alternatively, use -chdir argument:

{
  "args": [
    "-chdir=/path/to/terraform/config",
    "apply",
    "-auto-approve"
  ]
}

Step 2: Run the Configuration

Navigate to Run and Debug view (Ctrl+Shift+D)
Select “Run Terraform in debug mode”
Press F5 or click the green arrow

This executes the Terraform command just like running it from the command line, but with full debugging capabilities. Source: docs/debugging.md:74-98

Telemetry and Tracing

Terraform includes OpenTelemetry support for distributed tracing.

Enabling OpenTelemetry

Set the environment variable to enable OTLP trace exporter:

export OTEL_TRACES_EXPORTER=otlp

This is an experimental feature subject to change. Do not depend on the structure of trace output in production.

Implementation from telemetry.go:20-91:

// OpenTelemetry is disabled by default
func openTelemetryInit() error {
    if os.Getenv(openTelemetryExporterEnvVar) != "otlp" {
        return nil // Discard all telemetry by default
    }
    
    otelResource := resource.NewWithAttributes(
        semconv.SchemaURL,
        semconv.ServiceNameKey.String("Terraform CLI"),
        semconv.ServiceVersionKey.String(version.Version),
    )
    
    // Enable OTLP exporter with standard environment variables
    exp, err := autoexport.NewSpanExporter(context.Background())
    if err != nil {
        return err
    }
    
    sp := sdktrace.NewSimpleSpanProcessor(exp)
    provider := sdktrace.NewTracerProvider(
        sdktrace.WithSpanProcessor(sp),
        sdktrace.WithResource(otelResource),
    )
    otel.SetTracerProvider(provider)
    
    return nil
}

Standard OTLP Configuration

Configure the OTLP exporter using standard environment variables:

# Enable OTLP export
export OTEL_TRACES_EXPORTER=otlp

# Set OTLP endpoint
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

# Set OTLP headers (e.g., for authentication)
export OTEL_EXPORTER_OTLP_HEADERS="api-key=your-key-here"

# Set service name
export OTEL_SERVICE_NAME="terraform-debugging"

See OpenTelemetry OTLP Exporter Configuration for all options.

Using Tracers

Terraform initializes tracers in different packages:

// From internal/command/telemetry.go
var tracer trace.Tracer

func init() {
    tracer = otel.Tracer("github.com/hashicorp/terraform/internal/command")
}

Other packages with tracing:

internal/stacks/stackruntime/telemetry.go
internal/stacks/stackruntime/internal/stackeval/telemetry.go
internal/rpcapi/telemetry.go
internal/promising/telemetry.go

Logging

Environment Variables

Control logging verbosity with TF_LOG:

# Log everything
export TF_LOG=TRACE

# Log levels: TRACE, DEBUG, INFO, WARN, ERROR
export TF_LOG=DEBUG

# Log to file
export TF_LOG_PATH=terraform.log

# Component-specific logging
export TF_LOG_CORE=DEBUG
export TF_LOG_PROVIDER=TRACE

Reading Logs

Logs include:

Timestamp
Log level
Component
Message

Example:

2026-03-02T10:15:30.123-0700 [INFO]  backend/local: apply calling Apply
2026-03-02T10:15:30.456-0700 [DEBUG] backend/local: waiting for the ApplyGraph
2026-03-02T10:15:30.789-0700 [TRACE] vertex "provider.aws": visit complete

Common Debugging Scenarios

Debugging Cloud Backend Issues

Enable detailed HTTP logging:

export TF_LOG=DEBUG
export TF_LOG_PATH=cloud-debug.log
terraform plan

Check cloud-debug.log for:

API request/response details
Authentication issues
Service discovery problems
Version compatibility checks

Relevant code: internal/cloud/backend.go:611-619

Debugging Test Failures

Run test with verbose output:

terraform test -verbose -json > test-output.json

Set breakpoint in test runner:

Location: internal/command/test.go:193-200

Examine test state:

// In debugger, inspect:
// - preparation.Config: Loaded configuration
// - preparation.Variables: Input variables
// - runner: Test suite runner state
// - status: Current test status

Debugging Stacks Runtime

Set breakpoints in evaluation phases: Validation phase: Location: internal/stacks/stackruntime/internal/stackeval/ - Validate methods Planning phase: Location: internal/stacks/stackruntime/internal/stackeval/ - PlanChanges methods Apply phase: Location: internal/stacks/stackruntime/internal/stackeval/ - CheckApply methods Expression evaluation: Location: internal/stacks/stackruntime/internal/stackeval/ - EvalExpr functions

Debugging Provider Calls

Enable provider logging:

export TF_LOG_PROVIDER=TRACE
export TF_LOG_PATH=provider.log
terraform apply

Set breakpoints in provider communication:

Resource CRUD operations
Provider configuration
Schema validation

Performance Profiling

CPU Profiling

# Enable CPU profiling
export TF_CPU_PROFILE=cpu.prof
terraform apply

# Analyze profile
go tool pprof cpu.prof

Memory Profiling

# Enable memory profiling
export TF_MEM_PROFILE=mem.prof
terraform apply

# Analyze profile
go tool pprof mem.prof

Trace Profiling

# Generate execution trace
go test -trace=trace.out ./...

# View trace
go tool trace trace.out

Best Practices

1. Use Appropriate Debug Granularity

# For general debugging
export TF_LOG=DEBUG

# For deep investigation
export TF_LOG=TRACE

# For production troubleshooting
export TF_LOG=INFO

2. Isolate Issues with Minimal Configs

Create minimal reproduction:

# minimal.tf - Reproduce specific issue
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = "us-west-2"
}

resource "aws_instance" "test" {
  ami           = "ami-12345678"
  instance_type = "t2.micro"
}

3. Preserve Debug Information

Save debug output for analysis:

# Capture all output
terraform apply 2>&1 | tee terraform-debug.log

# With JSON output
terraform apply -json 2>&1 | tee terraform-debug.json

4. Use Conditional Breakpoints

In VS Code or Delve, set conditional breakpoints:

# Break only when specific condition is true
Condition: workspaceName == "production"

# Break after N hits
Hit Count: 10

5. Leverage Test Debugging

Write tests to reproduce issues:

// internal/command/test_debug_test.go
func TestReproduceIssue(t *testing.T) {
    // Set up minimal reproduction
    // Add breakpoint here
    // Step through issue
}

Debugging Resources

Documentation

Example Configurations

Terraform includes example debugging configurations:

docs/debugging-configs/vscode/debug-automated-tests/launch.json
docs/debugging-configs/vscode/launch-via-cli/launch.json
docs/debugging-configs/vscode/launch-from-vscode-debugger/launch.json

Testing Framework - Debug test failures
Cloud Integration - Debug remote operations
Stacks - Debug Stacks runtime issues

Getting Started

Core Concepts

Configuration Language

Workflow

State Management

Providers

Advanced Features

​Debugging Terraform

​Overview

​Debugging Automated Tests

​VS Code Test Debugging

​Custom Test Launch Configuration

​Debugging Terraform Operations

​Method 1: Delve CLI Tool

​Step 1: Compile with Debug Flags

​Step 2: Start Debug Server

​Step 3a: Connect via Delve CLI

​Step 3b: Connect via VS Code

​Method 2: VS Code Debugger (Direct Launch)

​Step 1: Create Launch Configuration

​Step 2: Run the Configuration

​Telemetry and Tracing

​Enabling OpenTelemetry

​Standard OTLP Configuration

​Using Tracers

​Logging

​Environment Variables

​Reading Logs

​Common Debugging Scenarios

​Debugging Cloud Backend Issues

​Debugging Test Failures

​Debugging Stacks Runtime

​Debugging Provider Calls

​Performance Profiling

​CPU Profiling

​Memory Profiling

​Trace Profiling

​Best Practices

​1. Use Appropriate Debug Granularity

​2. Isolate Issues with Minimal Configs

​3. Preserve Debug Information

​4. Use Conditional Breakpoints

​5. Leverage Test Debugging

​Debugging Resources

​Documentation

​Example Configurations

​Related Resources

Build docs developers (and LLMs) love

Debugging Terraform

Overview

Debugging Automated Tests

VS Code Test Debugging

Custom Test Launch Configuration

Debugging Terraform Operations

Method 1: Delve CLI Tool

Step 1: Compile with Debug Flags

Step 2: Start Debug Server

Step 3a: Connect via Delve CLI

Step 3b: Connect via VS Code

Method 2: VS Code Debugger (Direct Launch)

Step 1: Create Launch Configuration

Step 2: Run the Configuration

Telemetry and Tracing

Enabling OpenTelemetry

Standard OTLP Configuration

Using Tracers

Logging

Environment Variables

Reading Logs

Common Debugging Scenarios

Debugging Cloud Backend Issues

Debugging Test Failures

Debugging Stacks Runtime

Debugging Provider Calls

Performance Profiling

CPU Profiling

Memory Profiling

Trace Profiling

Best Practices

1. Use Appropriate Debug Granularity

2. Isolate Issues with Minimal Configs

3. Preserve Debug Information

4. Use Conditional Breakpoints

5. Leverage Test Debugging

Debugging Resources

Documentation

Example Configurations

Related Resources