Kubernetes + Tilt - Microservices App

Tilt provides a powerful local Kubernetes development experience with live updates, resource visualization, and integrated observability. This setup uses Kind (Kubernetes in Docker) for the cluster and Tilt for the development workflow.

Why Kubernetes + Tilt?

Compared to Docker Compose, this setup offers:

Production parity: Run the same manifests used in production
Observability: Prometheus, Grafana, Loki, and Tempo for metrics, logs, and traces
Network visibility: Hubble UI for visualizing service mesh traffic
Live updates: Fast rebuild and redeploy on code changes
Resource dashboard: Visual monitoring of all services in one place

This setup requires significantly more resources than Docker Compose. Recommended minimum: 16GB RAM, 4 CPU cores.

Prerequisites

Environment setup complete
Docker Desktop or Docker Engine running
At least 16GB RAM and 4 CPU cores available
Infrastructure repository cloned

Bootstrap Options

There are two bootstrap options depending on your available resources:

Full Bootstrap (Recommended)
Lightweight Bootstrap

Full Observability Stack

Includes Prometheus, Grafana, Loki, Tempo, Istio, ArgoCD, and Hubble UI.Requirements:

16GB+ RAM
4+ CPU cores
Fast storage (SSD recommended)

Clone infrastructure repository

git clone https://github.com/hackz-megalo-cup/microservices-infra
cd microservices-infra
direnv allow

Run full bootstrap

Ensure Docker is running, then:

full-bootstrap

This will:

Create a Kind cluster with multiple worker nodes
Install Cilium CNI with Hubble
Deploy Prometheus, Grafana, Loki, and Tempo
Install Istio service mesh
Deploy ArgoCD for GitOps
Configure Traefik as the ingress controller

The bootstrap process takes 10-15 minutes on first run. Subsequent runs are much faster.

Verify cluster is ready

kubectl get nodes
kubectl get pods -A

All pods should be in “Running” or “Completed” state.

Available Dashboards

After full bootstrap, access these UIs:

Service	URL	Credentials
Grafana	http://localhost:30300	admin / admin
Prometheus	http://localhost:30090	-
Hubble UI	http://localhost:31235	-
Tilt	http://localhost:10350	-
Traefik	http://localhost:30081	-

Minimal Kubernetes Setup

Runs Kubernetes without the full observability stack. Disables Istio and ArgoCD.Requirements:

8GB+ RAM
2+ CPU cores

Clone infrastructure repository

git clone https://github.com/hackz-megalo-cup/microservices-infra
cd microservices-infra
direnv allow

Run lightweight bootstrap

bootstrap

This will:

Create a Kind cluster with fewer nodes
Install Cilium CNI
Deploy Traefik ingress controller
Skip Prometheus, Grafana, Istio, and ArgoCD

You’ll still have Kubernetes features, just without the observability tools.

Verify cluster is ready

kubectl get nodes
kubectl get pods -A

Available Services

Service	URL
Tilt	http://localhost:10350
Traefik	http://localhost:30081

Starting Tilt

Once the cluster is bootstrapped, start the application services:

Navigate to application repository

cd /path/to/microservice-app

Start Tilt

Foreground (recommended for first run)
Background

Run Tilt in the foreground to watch startup logs:

tilt up

Press space to open the Tilt UI in your browser, or visit http://localhost:10350.Press q to stop Tilt and clean up resources.

Run Tilt in the background to free up your terminal:

tilt up > /dev/null 2>&1 &

Access the UI at http://localhost:10350.To stop:

tilt down

Monitor startup

Open http://localhost:10350 to see the Tilt dashboard.You’ll see resources building and deploying:

Bootstrap resources: Cluster setup, manifest generation
Codegen resources: Protocol Buffer compilation
Compile resources: Go service compilation (if not using Nix)
Kubernetes resources: Service deployments with port forwards

Access the application

Once all resources are green, access the frontend:

http://localhost:30081

Tilt Architecture

The Tiltfile orchestrates the following workflow:

┌─────────────────────────────────────────────────────┐
│                   Tilt Workflow                     │
└─────────────────────────────────────────────────────┘

1. Bootstrap Phase
   ├── cluster-up          Create Kind cluster (if needed)
   └── gen-manifests       Generate K8s manifests from nixidy

2. Code Generation Phase
   └── buf-generate        Generate Go/TypeScript from proto

3. Build Phase (if USE_NIX=false)
   ├── greeter-compile     Build Go binary
   ├── caller-compile      Build Go binary
   └── gateway-compile     Build Go binary

4. Image Phase
   ├── Go services         custom_build (Nix) or docker_build_with_restart
   ├── Node.js services    docker_build
   └── Frontend            docker_build with VITE_API_BASE_URL

5. Deploy Phase
   ├── Apply K8s manifests
   └── Set up port forwards

Resource Management

Tilt organizes resources into labeled groups:

Bootstrap Resources

cluster-up: Ensures Kind cluster exists
gen-manifests: Generates Kubernetes manifests from Nix modules

Codegen Resources

buf-generate: Compiles Protocol Buffers to Go and TypeScript

Compile Resources (non-Nix mode)

greeter-compile, caller-compile, gateway-compile: Build Go binaries

Kubernetes Resources

Each service runs as a Kubernetes Deployment:

Service	Port Forward	Dependencies
traefik	-	gen-manifests
caller-service	8081	gen-manifests, buf-generate, caller-compile
greeter-service	8080	gen-manifests, buf-generate, greeter-compile
gateway	8082	gen-manifests, buf-generate, gateway-compile
custom-lang-service	3000	gen-manifests
auth-service	8090	gen-manifests
frontend	5173	gen-manifests, buf-generate

Live Updates

Go Services (non-Nix mode)

Tilt uses docker_build_with_restart for fast iteration:

Edit Go source code
Local resource compiles binary (greeter-compile)
Binary is synced to running container via live_update
Container restarts with new binary

Live updates skip the full Docker build, making updates nearly instant.

Go Services (Nix mode)

With USE_NIX=true, Tilt uses Nix to build reproducible images:

export USE_NIX=true
tilt up

Edit Go source code
Nix builds a complete container image
Image is loaded into Kind cluster
Kubernetes pulls the new image

Nix builds are slower but provide exact reproducibility. Use for testing production builds.

Node.js Services

Node.js services use standard docker_build:

Edit Node.js source code
Tilt rebuilds the Docker image
Image is pushed to Kind
Kubernetes redeploys the pod

Frontend

Frontend uses docker_build with build args:

docker_build(
    'frontend',
    context='.',
    dockerfile='deploy/docker/frontend/Dockerfile',
    build_args={'VITE_API_BASE_URL': 'http://localhost:30081'},
)

For faster frontend iteration, use the local dev server instead of rebuilding Docker images.

Port Forwards

Tilt automatically forwards these ports to localhost:

Service	Local Port	Pod Port
greeter-service	8080	8080
caller-service	8081	8081
gateway	8082	8082
custom-lang-service	3000	3000
auth-service	8090	8090
frontend	5173	80

Additional infrastructure ports (from full bootstrap):

Service	Local Port
Traefik	30081
Grafana	30300
Prometheus	30090
Hubble UI	31235

Environment Variables

Tilt honors these environment variables:

`USE_NIX`

Default: false Use Nix to build container images for Go services:

export USE_NIX=true
tilt up

`TILT_SKIP_CLUSTER_UP`

Default: false Skip the cluster-up resource if you’ve already created the cluster manually:

export TILT_SKIP_CLUSTER_UP=true
tilt up

Debugging

Check resource status

In the Tilt UI (http://localhost:10350):

Green: Resource is healthy
Yellow: Resource is building/updating
Red: Resource failed

Click any resource to view logs and details.

Manual health check

Tilt includes a manual health check resource:

Open Tilt UI
Find health-check in the resource list
Click the trigger button

This runs:

kubectl cluster-info
kubectl get pods -A
kubectl get svc -A
curl http://localhost:8080/healthz   # greeter
curl http://localhost:8082/healthz   # gateway
curl http://localhost:5173           # frontend
curl http://localhost:30081          # traefik

Debug Kubernetes directly

Use the debug-k8s command from devenv:

debug-k8s

This shows pod status and recent events across all namespaces.

Debug gRPC services

debug-grpc

Tests greeter and gateway endpoints with grpcurl.

View logs

# Click on any resource in http://localhost:10350

Common Operations

Restart a service

In Tilt UI:

Find the service resource
Click the ⟳ (restart) button

Or via CLI:

kubectl rollout restart deployment/greeter-service

Trigger a rebuild

In Tilt UI:

Find the service resource
Click the ⚡ (rebuild) button

Or edit source code—Tilt auto-detects changes.

Stop Tilt

Foreground
Background

Press q in the terminal running tilt up.

tilt down

Destroy the cluster

kind delete cluster --name microservice-app

This deletes all Kubernetes resources and requires re-running bootstrap.

Updating Manifests

Kubernetes manifests are generated from Nix modules in deploy/nixidy/.

Edit manifest configuration

Edit files in deploy/nixidy/env/ or deploy/k8s/
Regenerate manifests:
```
gen-manifests
```
Tilt auto-detects changes and reapplies

Or use the watch mode:

watch-manifests

This watches nixidy files and auto-applies changes to the cluster.

Observability (Full Bootstrap Only)

Grafana Dashboards

Access Grafana at http://localhost:30300 (admin/admin). Pre-configured dashboards:

Kubernetes cluster metrics: Node and pod resource usage
Service metrics: Request rates, latencies, error rates
Logs (Loki): Aggregated logs from all services
Traces (Tempo): Distributed tracing for requests

Prometheus

Query metrics at http://localhost:30090. Example queries:

# Request rate per service
rate(http_requests_total[5m])

# Pod CPU usage
sum(rate(container_cpu_usage_seconds_total[1m])) by (pod)

Hubble UI

Visualize service mesh traffic at http://localhost:31235.

View real-time network flows
Inspect HTTP/gRPC requests
Debug connectivity issues

Performance Tips

Reduce resource usage

Use lightweight bootstrap instead of full
Disable unused services in Tiltfile
Limit Kind cluster resources in kind-config.yaml

Speed up builds

Use USE_NIX=false for faster Go builds
Leverage Docker build cache (avoid --no-cache)
Use live_update for instant Go binary sync

Optimize dependencies

In Tiltfile, carefully manage resource_deps to avoid unnecessary rebuilds:

# Good: Only rebuild when dependencies change
k8s_resource('greeter-service', resource_deps=['buf-generate', 'greeter-compile'])

# Bad: Rebuilds on every manifest change
k8s_resource('greeter-service', resource_deps=['gen-manifests'])

Troubleshooting

Tilt won’t start

Check cluster:

kubectl cluster-info

If cluster isn’t running:

cd microservices-infra
full-bootstrap  # or bootstrap

Resource stuck in yellow/red

Click the resource in Tilt UI
Check logs for errors
Common fixes:
- Run buf generate if proto changes aren’t detected
- Run gen-manifests if Nix changes aren’t applied
- Restart Tilt: tilt down && tilt up

Port forward failures

If a port is already in use:

# Find what's using the port
lsof -i :8080

# Kill the process or change port in Tiltfile

Manifests not found

If you see “No Kubernetes manifests found yet”:

# Ensure nixidy modules are committed to git
git add deploy/nixidy deploy/k8s

# Regenerate manifests
gen-manifests

# Restart Tilt
tilt down && tilt up

Nix only sees files tracked by git. Always git add new files before running gen-manifests.

Image pull errors

If Kubernetes can’t pull images:

# Load images manually
load-microservice-images

# Or rebuild in Tilt UI

Get Started

Development

Services

Protocol Buffers

Deployment

Guides

​Why Kubernetes + Tilt?

​Prerequisites

​Bootstrap Options

​Full Observability Stack

​Available Dashboards

​Minimal Kubernetes Setup

​Available Services

​Starting Tilt

​Tilt Architecture

​Resource Management

​Bootstrap Resources

​Codegen Resources

​Compile Resources (non-Nix mode)

​Kubernetes Resources

​Live Updates

​Go Services (non-Nix mode)

​Go Services (Nix mode)

​Node.js Services

​Frontend

​Port Forwards

​Environment Variables

​USE_NIX

​TILT_SKIP_CLUSTER_UP

​Debugging

​Check resource status

​Manual health check

​Debug Kubernetes directly

​Debug gRPC services

​View logs

​Common Operations

​Restart a service

​Trigger a rebuild

​Stop Tilt

​Destroy the cluster

​Updating Manifests

​Edit manifest configuration

​Observability (Full Bootstrap Only)

​Grafana Dashboards

​Prometheus

​Hubble UI

​Performance Tips

​Reduce resource usage

​Speed up builds

​Optimize dependencies

​Troubleshooting

​Tilt won’t start

​Resource stuck in yellow/red

​Port forward failures

​Manifests not found

​Image pull errors

​Next Steps

Frontend Development

Docker Compose

Build docs developers (and LLMs) love

Why Kubernetes + Tilt?

Prerequisites

Bootstrap Options

Full Observability Stack

Available Dashboards

Minimal Kubernetes Setup

Available Services

Starting Tilt

Tilt Architecture

Resource Management

Bootstrap Resources

Codegen Resources

Compile Resources (non-Nix mode)

Kubernetes Resources

Live Updates

Go Services (non-Nix mode)

Go Services (Nix mode)

Node.js Services

Frontend

Port Forwards

Environment Variables

`USE_NIX`

`TILT_SKIP_CLUSTER_UP`

Debugging

Check resource status

Manual health check

Debug Kubernetes directly

Debug gRPC services

View logs

Common Operations

Restart a service

Trigger a rebuild

Stop Tilt

Destroy the cluster

Updating Manifests

Edit manifest configuration

Observability (Full Bootstrap Only)

Grafana Dashboards

Prometheus

Hubble UI

Performance Tips

Reduce resource usage

Speed up builds

Optimize dependencies

Troubleshooting

Tilt won’t start

Resource stuck in yellow/red

Port forward failures

Manifests not found

Image pull errors

Next Steps