Production Deployment - Headscale + Tailscale Docker Stack

Overview

This guide covers deploying the Headscale stack to a production server with SSL/TLS certificates, security hardening, and automated certificate renewal.

Production deployment requires a domain name and proper SSL/TLS certificates. Do not expose Headscale without HTTPS in production.

Prerequisites

Infrastructure Requirements

Server: 2GB RAM minimum, 4GB recommended
Storage: 20GB minimum, 50GB recommended
OS: Ubuntu 22.04 LTS or similar Linux distribution
Docker: Version 20.10 or later
Docker Compose: Version 2.0 or later

Domain and DNS

Domain Name: You need a registered domain (e.g., headscale.example.com)
DNS Configuration: A record pointing to your server’s IP address
Firewall: Ports 80 and 443 must be open

Wait for DNS propagation before starting the SSL certificate setup. This can take 5 minutes to 24 hours depending on your DNS provider.

Initial Setup

Clone Repository

Clone the Headscale stack to your production server:

git clone <your-repo>
cd headscale-tailscale-docker

Configure Environment

Copy the environment template:

cp .env.example .env
nano .env

Configure production values:

.env

# Your actual domain name
HEADSCALE_DOMAIN=headscale.example.com

# PostgreSQL credentials
POSTGRES_DB=headscale
POSTGRES_USER=headscale
# Generate a strong password
POSTGRES_PASSWORD=<use-a-strong-random-password>

# Timezone
TZ=UTC

# Headplane credentials (generate after stack is running)
HEADPLANE_API_KEY=<generate-after-setup>
HEADPLANE_COOKIE_SECRET=<generate-with-openssl>

Use strong, randomly generated passwords. Never use default passwords in production.

Generate a strong password:

# Generate PostgreSQL password
openssl rand -base64 32

# Generate cookie secret
openssl rand -base64 24

Update Headscale Configuration

Edit config/config.yaml to match your domain:

config/config.yaml

# Update server URL
server_url: https://headscale.example.com

# Ensure database password matches .env
database:
  type: postgres
  postgres:
    host: postgres
    port: 5432
    name: headscale
    user: headscale
    password: <same-as-POSTGRES_PASSWORD-in-env>

The PostgreSQL password MUST match between .env (POSTGRES_PASSWORD) and config/config.yaml (database.postgres.password).

Configure Firewall

Open required ports:

# Allow HTTP and HTTPS
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp

# Allow SSH (if not already allowed)
sudo ufw allow 22/tcp

# Enable firewall
sudo ufw enable

# Check status
sudo ufw status

Verify DNS Configuration

Verify your DNS is configured correctly:

# Check if DNS resolves to your server
nslookup headscale.example.com

# Or use dig
dig headscale.example.com +short

The output should show your server’s IP address.

SSL/TLS Certificate Setup

The stack uses Let’s Encrypt with certbot for free SSL/TLS certificates.

Option 1: Using the Helper Script

Create Certificate Directories

mkdir -p certbot/conf certbot/www

Initialize SSL Certificates

./scripts/nginx.sh ssl-init

Follow the prompts to enter your email and domain name.

Verify Certificate

Check certificate information:

./scripts/nginx.sh ssl-info

Option 2: Manual Certificate Setup

Prepare Directories

mkdir -p certbot/conf certbot/www

Start nginx Temporarily

Start nginx to handle the ACME challenge:

docker compose up -d nginx

Obtain Certificate

Run certbot to obtain certificates:

docker compose run --rm certbot certonly \
  --webroot \
  --webroot-path=/var/www/certbot \
  --email [email protected] \
  --agree-tos \
  --no-eff-email \
  -d headscale.example.com

Replace [email protected] with your actual email address. This is used for certificate expiration notifications.

Restart nginx

Restart nginx to load the new certificates:

docker compose restart nginx

Start Production Stack

Do NOT create a docker-compose.override.yml file for production. The base docker-compose.yml is already configured for production.

Start All Services

docker compose up -d

Check Service Status

Verify all services are running:

docker compose ps

All services should show status “Up” or “healthy”.

View Logs

Monitor logs for any errors:

# View all logs
docker compose logs -f

# View specific service
docker compose logs -f headscale
docker compose logs -f nginx

Test HTTPS Endpoint

Verify the HTTPS endpoint is working:

curl https://headscale.example.com/health

Expected response:

{"status":"pass"}

Initial Configuration

Generate API Key

Create an API key for Headplane:

docker exec headscale headscale apikeys create --expiration 999d

Update .env with the generated key:

HEADPLANE_API_KEY=<your-generated-key>

Restart Headplane:

docker compose restart headplane

Create First User and Device

# Create a user
./scripts/headscale.sh users create admin

# Generate pre-auth key
./scripts/headscale.sh keys create admin --reusable --expiration 24h

Save the pre-auth key for connecting devices.

Connect Your First Device

Linux/macOS
Windows
Mobile (iOS/Android)

sudo tailscale up --login-server https://headscale.example.com --authkey YOUR_KEY --accept-routes

Open PowerShell as Administrator:

tailscale up --login-server https://headscale.example.com --authkey YOUR_KEY --accept-routes

Install Tailscale app from App Store or Google Play
Open the app
Tap “Use custom control server”
Enter: https://headscale.example.com
Tap “Continue”
Enter your pre-auth key when prompted

Production Architecture

Internet (Ports 80/443)
    |
    v
nginx Reverse Proxy
    ├── SSL/TLS Termination
    ├── Rate Limiting
    ├── Security Headers
    └── HTTP -> HTTPS Redirect
    |
    v
Headscale Server (:8080)
    ├── Control Plane API
    ├── WebSocket Support
    └── Health/Metrics
    |
    v
PostgreSQL Database
    └── Persistent Storage

Headplane UI (:3000)
    └── Web Management Interface

Certbot
    └── Automated SSL Renewal

Security Configuration

nginx Security Features

The production nginx.conf includes:

SSL/TLS: TLS 1.2 and 1.3 with strong cipher suites
HSTS: HTTP Strict Transport Security with preload
Rate Limiting: Protection against DDoS attacks
Security Headers: X-Frame-Options, CSP, X-Content-Type-Options
OCSP Stapling: Fast certificate validation

Rate Limiting Configuration

From nginx.conf:

# Rate limiting zones
limit_req_zone $binary_remote_addr zone=general:10m rate=10r/s;
limit_req_zone $binary_remote_addr zone=api:10m rate=30r/s;
limit_req_zone $binary_remote_addr zone=health:10m rate=100r/s;

# Connection limit per IP
limit_conn_zone $binary_remote_addr zone=addr:10m;

Adjust these values based on your traffic patterns.

Firewall Rules

Minimal firewall configuration:

# Allow only essential ports
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22/tcp    # SSH
sudo ufw allow 80/tcp    # HTTP (for ACME challenges)
sudo ufw allow 443/tcp   # HTTPS
sudo ufw enable

Restrict Metrics Endpoint

Uncomment in nginx.conf to restrict metrics access:

location = /metrics {
    # Restrict to internal networks only
    allow 127.0.0.1;
    allow 10.0.0.0/8;
    allow 172.16.0.0/12;
    allow 192.168.0.0/16;
    deny all;

    proxy_pass http://headscale_backend/metrics;
}

Production Maintenance

Certificate Renewal

Certificates are automatically renewed by certbot every 12 hours. Check certificate expiration:

./scripts/nginx.sh ssl-info

Manual renewal (if needed):

docker compose run --rm certbot renew
docker compose restart nginx

Force renewal (testing):

docker compose run --rm certbot renew --force-renewal

Database Backups

Create automated backups:

# Manual backup
./scripts/backup.sh

# Backups are saved to ./backups/
ls -lh backups/

Backup script contents:

# PostgreSQL backup
docker exec headscale-db pg_dump -U headscale headscale > backup.sql

# Full system backup
tar -czf headscale-backup-$(date +%Y%m%d).tar.gz config/ data/ headplane/

Setup automated backups with cron:

# Edit crontab
crontab -e

# Add daily backup at 2 AM
0 2 * * * cd /path/to/headscale-tailscale-docker && ./scripts/backup.sh

Updates

Update Docker images:

# Pull latest images
docker compose pull

# Restart with new images
docker compose up -d

# Check logs
docker compose logs -f

Monitoring

Health Checks
Logs
Metrics

# Service health
./scripts/nginx.sh health

# Individual service status
docker compose ps

# Test HTTPS endpoint
curl https://headscale.example.com/health

# nginx access logs
./scripts/nginx.sh access-logs 20

# nginx error logs
./scripts/nginx.sh error-logs 20

# Follow all logs
docker compose logs -f

# View Prometheus metrics
curl http://localhost:9090/metrics

# Resource usage
docker stats

# Disk usage
du -sh data/ postgres-data/

Troubleshooting

SSL Certificate Issues

Certificate not found:

# Check if certificates exist
ls -la certbot/conf/live/headscale.example.com/

# View certbot logs
docker compose logs certbot

# Try obtaining certificate again
docker compose run --rm certbot certonly --webroot \
  --webroot-path=/var/www/certbot \
  --email [email protected] \
  --agree-tos \
  -d headscale.example.com

Certificate expired:

# Check expiration
./scripts/nginx.sh ssl-info

# Force renewal
docker compose run --rm certbot renew --force-renewal
docker compose restart nginx

nginx Won’t Start

# Test configuration
./scripts/nginx.sh test

# Check error logs
./scripts/nginx.sh error-logs 50

# Verify ports not in use
sudo lsof -i :80
sudo lsof -i :443

Database Connection Failed

Verify password synchronization between .env and config/config.yaml.

# Check database health
docker compose exec postgres pg_isready

# Test connection
docker exec -it headscale-db psql -U headscale -c "SELECT 1;"

# View database logs
docker compose logs postgres

Performance Issues

# Check resource usage
docker stats

# Check disk space
df -h

# Check database size
docker exec -it headscale-db psql -U headscale -c "\l+"

# Vacuum database
docker exec -it headscale-db psql -U headscale -c "VACUUM ANALYZE;"

Switching from Development to Production

Stop Development Stack

docker compose down

Remove Override File

# Remove development override
rm docker-compose.override.yml

Update Environment

nano .env
# Change HEADSCALE_DOMAIN to your production domain

Update Headscale Config

nano config/config.yaml
# Change server_url to https://your-domain.com

Setup SSL Certificates

./scripts/nginx.sh ssl-init

Start Production Stack

docker compose up -d

Best Practices

Regular Backups

Set up automated daily backups using cron jobs

Monitor Logs

Regularly review logs for errors and suspicious activity

Update Regularly

Keep Docker images and OS packages up to date

Strong Passwords

Use strong, randomly generated passwords for all services

Next Steps

Docker Compose Reference

Learn about all Docker Compose services and configuration

Local Development

Set up a local development environment

Get Started

Deployment

Configuration

Guides

Operations

​Overview

​Prerequisites

​Infrastructure Requirements

​Domain and DNS

​Initial Setup

​SSL/TLS Certificate Setup

​Option 1: Using the Helper Script

​Option 2: Manual Certificate Setup

​Start Production Stack

​Initial Configuration

​Generate API Key

​Create First User and Device

​Connect Your First Device

​Production Architecture

​Security Configuration

​nginx Security Features

​Rate Limiting Configuration

​Firewall Rules

​Restrict Metrics Endpoint

​Production Maintenance

​Certificate Renewal

​Database Backups

​Updates

​Monitoring

​Troubleshooting

​SSL Certificate Issues

​nginx Won’t Start

​Database Connection Failed

​Performance Issues

​Switching from Development to Production

​Best Practices

Regular Backups

Monitor Logs

Update Regularly

Strong Passwords

​Next Steps

Docker Compose Reference

Local Development

Build docs developers (and LLMs) love

Overview

Prerequisites

Infrastructure Requirements

Domain and DNS

Initial Setup

SSL/TLS Certificate Setup

Option 1: Using the Helper Script

Option 2: Manual Certificate Setup

Start Production Stack

Initial Configuration

Generate API Key

Create First User and Device

Connect Your First Device

Production Architecture

Security Configuration

nginx Security Features

Rate Limiting Configuration

Firewall Rules

Restrict Metrics Endpoint

Production Maintenance

Certificate Renewal

Database Backups

Updates

Monitoring

Troubleshooting

SSL Certificate Issues

nginx Won’t Start

Database Connection Failed

Performance Issues

Switching from Development to Production

Best Practices

Next Steps