Skip to main content
This guide covers debugging techniques and tools for troubleshooting issues in your local development environment.

​
Quick Debug Commands

The project provides several debugging commands in the devenv shell: 
# Check pod status and recent events
debug-k8s

​
Kubernetes Debugging

​
Using debug-k8s

The debug-k8s command provides a quick overview of your cluster state: 
debug-k8s
This shows:
  • Pod status across all namespaces
  • Recent Kubernetes events (last 10)
Example output: 
=== Pod status ===
NAMESPACE         NAME                                    READY   STATUS    RESTARTS
microservices     greeter-service-7d8f9c8b4d-x2k9j       1/1     Running   0
microservices     caller-service-5b6c7d8e9f-p3m4n        1/1     Running   0
microservices     gateway-6c7d8e9f0a-q4n5o               1/1     Running   0

=== Recent events ===
5m          Normal    Scheduled    pod/greeter-service-...    Successfully assigned
4m          Normal    Pulled       pod/greeter-service-...    Container image pulled
4m          Normal    Created      pod/greeter-service-...    Created container
4m          Normal    Started      pod/greeter-service-...    Started container

​
Manual Kubernetes Debugging

# Follow logs from a specific pod
kubectl -n microservices logs -f deploy/greeter-service

# View logs from all containers in a pod
kubectl -n microservices logs -f pod-name --all-containers

# View previous container logs (after crash)
kubectl -n microservices logs deploy/greeter-service --previous

# Get detailed pod information
kubectl -n microservices describe pod greeter-service-7d8f9c8b4d-x2k9j

# Check deployment status
kubectl -n microservices describe deploy greeter-service

# Open a shell in the pod
kubectl -n microservices exec -it deploy/greeter-service -- /bin/sh

# Run a single command
kubectl -n microservices exec deploy/greeter-service -- env

# List services
kubectl -n microservices get svc

# Describe service
kubectl -n microservices describe svc greeter-service

# Check endpoints
kubectl -n microservices get endpoints greeter-service

# Forward local port to service
kubectl -n microservices port-forward svc/greeter-service 8080:80

# Forward to specific pod
kubectl -n microservices port-forward pod/greeter-service-xxx 8080:8080

# Multiple ports
kubectl -n microservices port-forward svc/gateway 8082:8082 8083:8083

​
gRPC Debugging

​
Using debug-grpc

The debug-grpc command tests gRPC connectivity: 
debug-grpc
This runs: 
# List available services on greeter
grpcurl -plaintext localhost:8080 list

# List available services on gateway
grpcurl -plaintext localhost:8082 list
Example output: 
=== Greeter gRPC check ===
greeter.v1.GreeterService
grpc.health.v1.Health
grpc.reflection.v1alpha.ServerReflection

=== Gateway gRPC check ===
gateway.v1.GatewayService
grpc.health.v1.Health
grpc.reflection.v1alpha.ServerReflection

​
Manual grpcurl Usage

1

List available services

grpcurl -plaintext localhost:8080 list
2

List methods in a service

grpcurl -plaintext localhost:8080 list greeter.v1.GreeterService
Output:
greeter.v1.GreeterService.Greet
3

Describe a method

grpcurl -plaintext localhost:8080 describe greeter.v1.GreeterService.Greet
Output:
greeter.v1.GreeterService.Greet is a method:
rpc Greet ( .greeter.v1.GreetRequest ) returns ( .greeter.v1.GreetResponse );
4

Describe message types

grpcurl -plaintext localhost:8080 describe greeter.v1.GreetRequest
Output:
greeter.v1.GreetRequest is a message:
message GreetRequest {
  string name = 1;
}
5

Call a method with JSON

grpcurl -plaintext -d '{"name": "World"}' \
  localhost:8080 greeter.v1.GreeterService/Greet
Output:
{
  "message": "Hello World from greeter-service!",
  "externalStatus": 200,
  "externalBodyLength": 42
}
6

Call with headers

grpcurl -plaintext \
  -H "authorization: Bearer eyJhbGc..." \
  -d '{"name": "Authenticated"}' \
  localhost:8080 greeter.v1.GreeterService/Greet

​
Common grpcurl Examples

# Basic greet
grpcurl -plaintext -d '{"name": "Alice"}' \
  localhost:8080 greeter.v1.GreeterService/Greet

# Empty name (should default to "World")
grpcurl -plaintext -d '{}' \
  localhost:8080 greeter.v1.GreeterService/Greet

​
Service-Specific Debugging

​
Go Services

# Docker Compose
docker compose logs -f greeter

# Kubernetes
kubectl -n microservices logs -f deploy/greeter-service

# Direct to service
curl http://localhost:8080/healthz

# Via Traefik
curl http://localhost:30081/healthz

# Docker Compose
docker compose exec greeter env

# Kubernetes
kubectl -n microservices exec deploy/greeter-service -- env
Add to docker-compose.yml:
greeter:
  command: ["dlv", "debug", "--headless", "--listen=:2345", "--api-version=2"]
  ports:
    - "2345:2345"
Connect with your IDE or:
dlv connect localhost:2345

​
Node.js Services

# Docker Compose
docker compose logs -f auth-service

# Kubernetes
kubectl -n microservices logs -f deploy/auth-service

curl http://localhost:8090/healthz

# Login
TOKEN=$(curl -s -X POST http://localhost:8090/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"[email protected]","password":"password"}' \
  | jq -r '.token')

echo "Token: $TOKEN"

# Verify token
curl http://localhost:8090/verify \
  -H "Authorization: Bearer $TOKEN"
Add to docker-compose.yml:
auth-service:
  command: ["node", "--inspect=0.0.0.0:9229", "server.js"]
  ports:
    - "9229:9229"
Connect with Chrome DevTools:
chrome://inspect

​
Docker Compose Debugging

# List running containers
docker compose ps

# View resource usage
docker compose stats

# View container details
docker compose ps greeter

​
Tilt Debugging

​
Tilt Dashboard

Access the Tilt UI at http://localhost:10350 The dashboard shows:
  • Build status for each service
  • Pod status and logs
  • Resource dependencies
  • Error messages and warnings

​
Tilt Commands

# View Tilt logs
tilt logs

# Logs for specific resource
tilt logs greeter-service

# Trigger rebuild
tilt trigger greeter-service

# Disable resource
tilt disable greeter-service

# Enable resource
tilt enable greeter-service

# Get resource status
tilt get all

​
Common Issues and Solutions

Check logs:
kubectl -n microservices logs deploy/greeter-service --previous
Common causes:
  • Missing environment variables
  • Port already in use
  • Failed health check
  • Panic in Go code
Solution:
# Check pod events
kubectl -n microservices describe pod <pod-name>

# Verify environment
kubectl -n microservices get deploy greeter-service -o yaml | grep -A 20 env:
Symptoms:
Failed to dial target host "localhost:8080": context deadline exceeded
Check service is running:
# Docker Compose
docker compose ps greeter

# Kubernetes with port-forward
kubectl -n microservices port-forward svc/greeter-service 8080:80
Verify port:
# Check what's listening
lsof -i :8080
netstat -an | grep 8080
Check network:
# Docker Compose - verify network
docker compose exec greeter ping caller

# Kubernetes - verify service discovery
kubectl -n microservices exec deploy/greeter-service -- nslookup caller-service
Verify service endpoints:
kubectl -n microservices get endpoints
Error: file not foundSolution: Git add new files (Nix only sees tracked files)
git add deploy/k8s/my-service.nix
git add proto/my-service/
Error: chartHash mismatchSolution:
fix-chart-hash
Check Traefik dashboard:
  • Docker Compose: http://localhost:8080 (if enabled)
  • Kubernetes: kubectl -n microservices port-forward svc/traefik 8080:80
Verify labels:
# Docker Compose
docker compose config | grep -A 10 labels

# Kubernetes
kubectl -n microservices get ingress
Test without Traefik:
# Port-forward directly to service
kubectl -n microservices port-forward svc/greeter-service 8080:80
grpcurl -plaintext localhost:8080 list

​
Monitoring and Observability

​
Accessing Monitoring Tools

When running with full-bootstrap:

Grafana

URL: http://localhost:30300
Credentials: admin / admin
View metrics, logs (Loki), and traces (Tempo)

Prometheus

URL: http://localhost:30090
Query raw metrics and check targets

Hubble UI

URL: http://localhost:31235
Visualize network flows between services

Tilt Dashboard

URL: http://localhost:10350
Build status, logs, and resource management

​
See Also

Build docs developers (and LLMs) love

Get started for freeTalk to us