Updates and Maintenance - Headscale + Tailscale Docker Stack

Overview

Regular updates ensure your Headscale deployment remains secure, stable, and feature-rich. This guide covers version management, update procedures, rollback strategies, and ongoing maintenance.

Headscale uses semantic versioning (vX.Y.Z). This stack pins to specific versions to prevent unexpected breaking changes during automatic updates.

Version Management

Current Versions

From docker-compose.yml:

services:
  headscale:
    image: headscale/headscale:v0.27.0  # Pinned version
  
  postgres:
    image: postgres:18-alpine
  
  nginx:
    image: nginx:alpine
  
  headplane:
    image: ghcr.io/tale/headplane:latest

Why Pin Versions?

DO: Pin Versions

image: headscale/headscale:v0.27.0

Predictable updates, test before deploying, avoid breaking changes

DON'T: Use Latest

image: headscale/headscale:latest

Unpredictable updates, potential breaking changes, difficult rollback

Checking for Updates

Headscale Updates

# Check current version
docker exec headscale headscale version

# Check for new releases
curl -s https://api.github.com/repos/juanfont/headscale/releases/latest | \
  grep '"tag_name"' | \
  cut -d'"' -f4

# Compare versions
CURRENT=$(docker exec headscale headscale version | grep -o 'v[0-9]\+\.[0-9]\+\.[0-9]\+')
LATEST=$(curl -s https://api.github.com/repos/juanfont/headscale/releases/latest | grep '"tag_name"' | cut -d'"' -f4)
echo "Current: $CURRENT"
echo "Latest: $LATEST"

Check All Services

# View current images
docker compose images

# Check for updates (doesn't download)
docker compose pull --dry-run

# Pull new images
docker compose pull

headscale: Pulling from headscale/headscale
digest: sha256:abc123... (new)
Status: Downloaded newer image for headscale/headscale:v0.27.0

Update Procedures

Pre-Update Checklist

Backup Everything

# Run backup script
./scripts/backup.sh

# Or manual backup
tar -czf pre-update-backup-$(date +%Y%m%d).tar.gz \
  config/ data/ .env headplane/

Check Changelog

# View release notes
open https://github.com/juanfont/headscale/releases

# Look for:
# - Breaking changes
# - Database migrations
# - Configuration changes
# - Deprecated features

Test in Staging

If possible, test the update in a staging environment first:

# Clone production config to staging
cp -r /production/headscale /staging/headscale-test
cd /staging/headscale-test

# Update version
nano docker-compose.yml

# Test update
docker compose pull
docker compose up -d

Schedule Maintenance Window

Notify users of brief downtime (typically 2-5 minutes):

# Example notification
echo "Headscale maintenance scheduled for $(date -d '+1 hour' '+%Y-%m-%d %H:%M')"

Standard Update Process

Minor/Patch Updates
Major Updates
All Services

For updates within the same major version (e.g., v0.27.0 → v0.27.1):

# 1. Backup
./scripts/backup.sh

# 2. Update version in docker-compose.yml
nano docker-compose.yml
# Change: image: headscale/headscale:v0.27.1

# 3. Pull new image
docker compose pull headscale

# 4. Recreate container
docker compose up -d headscale

# 5. Verify
docker exec headscale headscale version
curl http://localhost:8000/health

# 6. Check logs
docker compose logs -f headscale

Downtime: ~10-30 seconds

For major version updates (e.g., v0.27.0 → v1.0.0):

# 1. READ CHANGELOG CAREFULLY
# Major updates often include breaking changes

# 2. Backup EVERYTHING
docker compose stop headscale
tar -czf major-update-backup-$(date +%Y%m%d).tar.gz \
  config/ data/ .env headplane/
docker compose start headscale

# 3. Update configuration if needed
# Check for deprecated options
nano config/config.yaml

# 4. Update docker-compose.yml
nano docker-compose.yml

# 5. Pull and restart
docker compose pull
docker compose up -d

# 6. Monitor logs closely
docker compose logs -f headscale | tee update-$(date +%Y%m%d).log

# 7. Test functionality
docker exec headscale headscale nodes list
docker exec headscale headscale users list

# 8. Test node connectivity
# From a connected node:
tailscale status
ping <another-node-ip>

Downtime: 1-5 minutes

Update all services at once:

# 1. Backup
./scripts/backup.sh

# 2. Review changelogs for all services
# - Headscale
# - PostgreSQL
# - nginx
# - Headplane

# 3. Update versions in docker-compose.yml
nano docker-compose.yml

# 4. Pull all new images
docker compose pull

# 5. Recreate all containers
docker compose up -d

# 6. Watch logs
docker compose logs -f

# 7. Verify all services
docker compose ps
curl http://localhost:8000/health
curl http://localhost:3001/admin/

Downtime: 2-5 minutes

Database Migrations

Headscale automatically handles database migrations:

# Watch migration logs during update
docker compose logs -f headscale | grep -i migration

# Successful migration output:
# INFO migration: running migration 001_initial
# INFO migration: migration 001_initial completed

Database migrations are irreversible. Always backup before updating, especially for major version changes.

Rollback Procedures

Quick Rollback

If the update causes issues:

# 1. Stop services immediately
docker compose down

# 2. Restore previous version
nano docker-compose.yml
# Change back to: image: headscale/headscale:v0.27.0

# 3. Start with old version
docker compose up -d

# 4. Verify
curl http://localhost:8000/health
docker compose logs headscale

Full Restoration

If configuration or database changed:

# 1. Stop all services
docker compose down

# 2. Restore from backup
tar -xzf pre-update-backup-YYYYMMDD.tar.gz

# 3. Verify restored files
ls -la config/ data/

# 4. Start services
docker compose up -d

# 5. Verify restoration
docker exec headscale headscale version
docker exec headscale headscale nodes list

PostgreSQL Rollback

If database migration failed:

# 1. Stop Headscale
docker compose stop headscale

# 2. Restore database
cat backups/database_TIMESTAMP.sql | \
  docker exec -i headscale-db psql -U headscale

# 3. Revert to previous Headscale version
nano docker-compose.yml

# 4. Restart
docker compose up -d

Update Strategies

Automatic Updates (Not Recommended)

Automatic updates can introduce breaking changes. Use only for non-production environments.

docker-compose.yml

services:
  headscale:
    image: headscale/headscale:latest  # NOT RECOMMENDED for production

Manual Updates (Recommended)

# Monthly update schedule
# Check for updates
curl -s https://api.github.com/repos/juanfont/headscale/releases/latest

# Review changelog
open https://github.com/juanfont/headscale/releases

# Backup, update, test
./scripts/backup.sh
# Update docker-compose.yml
docker compose pull && docker compose up -d

Staged Rollout

For large deployments:

Test Environment

# Deploy to test environment first
cd /staging/headscale
docker compose pull
docker compose up -d

# Run tests
./scripts/run-tests.sh

Canary Deployment

# Update one production node
cd /production/headscale-01
docker compose pull
docker compose up -d

# Monitor for 24-48 hours
watch -n 300 'docker compose logs --tail 50 headscale | grep -i error'

Full Rollout

# If canary successful, update all nodes
for node in /production/headscale-*; do
  cd $node
  docker compose pull
  docker compose up -d
  sleep 60  # Stagger updates
done

Maintenance Tasks

Regular Maintenance Schedule

Daily
Weekly
Monthly
Quarterly

# Automated via cron
# Check health
0 */6 * * * curl -sf http://localhost:8000/health || echo "Health check failed"

# Backup
0 2 * * * cd /path/to/headscale && ./scripts/backup.sh

# Check for updates
docker compose pull --dry-run

# Review logs
docker compose logs --since 168h | grep -i error

# Check disk usage
df -h
docker system df

# Clean up old backups
find backups/ -name "*.sql" -mtime +7 -delete

# Apply updates
./scripts/backup.sh
docker compose pull
docker compose up -d

# Database maintenance
docker exec headscale-db vacuumdb -U headscale -d headscale -z

# Clean up expired keys
docker exec headscale headscale preauthkeys list

# Review and update ACLs
nano config/policy.json

# Check certificate expiration
openssl x509 -in certbot/conf/live/*/fullchain.pem -noout -dates

# Test backup restoration
mkdir -p /tmp/restore-test
cd /tmp/restore-test
tar -xzf /path/to/backup.tar.gz
docker compose up -d
# Verify and clean up

# Security audit
docker compose exec headscale headscale apikeys list
docker compose exec headscale headscale nodes list

# Review access logs
grep -E "401|403|500" logs/nginx/access.log

# Update documentation
# - Network topology
# - ACL policies
# - Contact information

Cleanup Operations

# Docker cleanup
docker image prune -a     # Remove unused images
docker volume prune       # Remove unused volumes
docker system prune -a    # Full cleanup

# Log cleanup
find logs/ -name "*.log" -mtime +30 -delete
: > logs/nginx/access.log  # Truncate large logs

# Database cleanup
docker exec headscale headscale nodes expire --all-offline

# Expire old pre-auth keys
docker exec headscale headscale preauthkeys expire --user myuser --key KEY

Performance Optimization

# Optimize PostgreSQL
docker exec headscale-db vacuumdb -U headscale -d headscale --analyze --verbose

# Rebuild indexes
docker exec headscale-db psql -U headscale -c "REINDEX DATABASE headscale;"

# Check query performance
docker exec headscale-db psql -U headscale -c \
  "SELECT query, calls, mean_exec_time FROM pg_stat_statements ORDER BY mean_exec_time DESC LIMIT 10;"

Monitoring Updates

Update Notifications

Set up notifications for new releases:

monitor-updates.sh

#!/bin/bash

CURRENT="v0.27.0"
LATEST=$(curl -s https://api.github.com/repos/juanfont/headscale/releases/latest | grep '"tag_name"' | cut -d'"' -f4)

if [ "$CURRENT" != "$LATEST" ]; then
    echo "New Headscale version available: $LATEST" | \
      mail -s "Headscale Update Available" [email protected]
fi

cron

# Check daily at 9 AM
0 9 * * * /path/to/monitor-updates.sh

GitHub Watch

Get notified of new releases:

Visit https://github.com/juanfont/headscale
Click “Watch” → “Custom” → “Releases”
Enable email notifications

Troubleshooting Updates

Update fails with migration error

# Check migration logs
docker compose logs headscale | grep migration

# If migration failed, rollback
docker compose down
tar -xzf pre-update-backup.tar.gz
docker compose up -d

# Contact support with logs

Nodes disconnect after update

# Check server_url hasn't changed
grep server_url config/config.yaml

# Verify health
curl http://localhost:8000/health

# Force client reconnection
# On each client:
tailscale down && tailscale up

Headplane stops working after update

# Check API compatibility
docker compose logs headplane | grep -i error

# Update Headplane
docker compose pull headplane
docker compose up -d headplane

# Regenerate API key if needed
docker exec headscale headscale apikeys create --expiration 999d

Best Practices

Always Backup

Backup before every update, especially for major versions

Read Changelogs

Review release notes for breaking changes and new features

Test First

Test updates in staging before production deployment

Pin Versions

Use specific versions, not latest tag

Schedule Maintenance

Notify users of planned downtime

Monitor After

Watch logs and metrics after updates

Document Changes

Keep changelog of updates and configuration changes

Keep Dependencies Updated

Update PostgreSQL, nginx, and Headplane regularly

Security Updates

Critical Security Patches

For urgent security updates:

# 1. Review security advisory
open https://github.com/juanfont/headscale/security/advisories

# 2. Immediate backup
./scripts/backup.sh

# 3. Apply update immediately
nano docker-compose.yml  # Update version
docker compose pull
docker compose up -d

# 4. Verify patch applied
docker exec headscale headscale version

# 5. Monitor for issues
docker compose logs -f headscale

Security patches should be applied as soon as possible, even outside regular maintenance windows.

Backup & Restore

Backup procedures before updates

Monitoring

Monitor health after updates

Troubleshooting

Resolve update issues

Security

Security best practices

Get Started

Deployment

Configuration

Guides

Operations

​Overview

​Version Management

​Current Versions

​Why Pin Versions?

DO: Pin Versions

DON'T: Use Latest

​Checking for Updates

​Headscale Updates

​Check All Services

​Update Procedures

​Pre-Update Checklist

​Standard Update Process

​Database Migrations

​Rollback Procedures

​Quick Rollback

​Full Restoration

​PostgreSQL Rollback

​Update Strategies

​Automatic Updates (Not Recommended)

​Manual Updates (Recommended)

​Staged Rollout

​Maintenance Tasks

​Regular Maintenance Schedule

​Cleanup Operations

​Performance Optimization

​Monitoring Updates

​Update Notifications

​GitHub Watch

​Troubleshooting Updates

​Best Practices

Always Backup

Read Changelogs

Test First

Pin Versions

Schedule Maintenance

Monitor After

Document Changes

Keep Dependencies Updated

​Security Updates

​Critical Security Patches

​Related Resources

Backup & Restore

Monitoring

Troubleshooting

Security

Build docs developers (and LLMs) love

Overview

Version Management

Current Versions

Why Pin Versions?

Checking for Updates

Headscale Updates

Check All Services

Update Procedures

Pre-Update Checklist

Standard Update Process

Database Migrations

Rollback Procedures

Quick Rollback

Full Restoration

PostgreSQL Rollback

Update Strategies

Automatic Updates (Not Recommended)

Manual Updates (Recommended)

Staged Rollout

Maintenance Tasks

Regular Maintenance Schedule

Cleanup Operations

Performance Optimization

Monitoring Updates

Update Notifications

GitHub Watch

Troubleshooting Updates

Best Practices

Security Updates

Critical Security Patches

Related Resources