Monitoring - Uncloud

Monitoring your Uncloud cluster helps you understand service health, diagnose issues, and ensure your applications run smoothly.

Service status

List all services

View all running services in your cluster:

uc service ls

Example output:

NAME       MODE         REPLICAS   IMAGE              ENDPOINTS
web-api    replicated   3          image/api:v2.1     https://api.example.com → :8000
caddy      global       2          caddy:2.9.1        
worker     replicated   1          image/worker:v1    

The output shows:

NAME: Service name
MODE: Deployment mode (replicated or global)
REPLICAS: Number of running containers
IMAGE: Container image(s) used
ENDPOINTS: Published ports and domains

Inspect a specific service

Get detailed information about a service:

uc service inspect web-api

Example output:

Service ID: 3f9a4b2c
Name:       web-api
Mode:       replicated

CONTAINER ID   IMAGE            CREATED          STATUS              IP ADDRESS    MACHINE
a7f3           image/api:v2.1   2 hours ago      Up 2 hours          10.210.0.5    oracle-vm
b8e2           image/api:v2.1   2 hours ago      Up 2 hours          10.210.1.8    hetzner-server
c9d1           image/api:v2.1   2 hours ago      Up 2 hours          10.210.2.12   home-server

This shows:

Service metadata (ID, name, mode)
All containers running as part of this service
Container state (Up, Exited, Restarting, etc.)
IP addresses and machine assignments

Quick status with `uc ps`

List all running containers across your cluster:

uc ps

This provides a Docker-like view of all containers, useful for a quick health check.

Container logs

View service logs

Stream logs from all replicas of a service:

uc service logs web-api

Example output:

Jan 15 14:23:45 oracle-vm      web-api[a7f3]  [INFO] Server started on port 8000
Jan 15 14:23:45 hetzner-server web-api[b8e2]  [INFO] Server started on port 8000
Jan 15 14:23:47 home-server    web-api[c9d1]  [INFO] Server started on port 8000
Jan 15 14:24:12 oracle-vm      web-api[a7f3]  [INFO] GET /api/users 200 45ms

Logs are merged from all replicas across machines, sorted by timestamp. Each line shows:

Timestamp
Machine name
Service name and container ID
Log message

Follow logs in real-time

Stream logs as they happen:

uc service logs -f web-api

Press Ctrl+C to stop streaming.

Limit log lines

Show the last 20 lines per replica:

uc service logs -n 20 web-api

Show all logs without limit:

uc service logs -n all web-api

Time-based filtering

View logs from the last 2 hours:

uc service logs --since 2h web-api

View logs from a specific time range:

uc service logs --since 2024-01-15T10:00:00 --until 2024-01-15T12:00:00 web-api

Supported time formats:

Relative duration: 2m30s, 1h, 3h30m
RFC 3339 date: 2024-01-15, 2024-01-15T10:30:00
RFC 3339 with timezone: 2024-01-15T10:30:00Z
Unix timestamp: 1763953966

Filter by machine

View logs from specific machines:

uc service logs -m oracle-vm web-api

Multiple machines:

uc service logs -m oracle-vm -m hetzner-server web-api

View logs for multiple services

Stream logs from multiple services:

uc service logs web-api worker

Services are color-coded for easy identification.

View logs from all services

If you have a compose.yaml file, view logs from all defined services:

uc service logs

This is useful for monitoring your entire application stack.

UTC timestamps

Show timestamps in UTC instead of local timezone:

uc service logs --utc web-api

System logs

For deeper system-level debugging, SSH into machines and check systemd logs.

Uncloud daemon logs

View the Uncloud daemon logs:

ssh [email protected]
sudo journalctl -u uncloud -f

The daemon logs show:

Container lifecycle events
gRPC API calls
WireGuard network changes
Cluster state updates

Corrosion logs

View the Corrosion (distributed database) logs:

sudo journalctl -u uncloud-corrosion -f

Corrosion logs show:

State replication events
Database operations
Gossip protocol messages

Docker logs

Check Docker daemon logs:

sudo journalctl -u docker -f

Useful for debugging Docker-specific issues.

System logs

View general system logs:

sudo journalctl -f

Add filters:

# Last hour
sudo journalctl --since "1 hour ago"

# Specific service
sudo journalctl -u uncloud --since today

# By priority (error and above)
sudo journalctl -p err -f

Health check monitoring

Uncloud doesn’t have built-in health checks yet, but you can monitor service health using:

Container state

Check if containers are running:

uc service inspect web-api

Look for the STATUS column. Containers should show “Up X hours/minutes”.

Health check scripts

Create a simple health check script:

#!/bin/bash
# health-check.sh

SERVICE="web-api"
STATUS=$(uc service inspect $SERVICE 2>&1)

if echo "$STATUS" | grep -q "Up"; then
  echo "✓ $SERVICE is healthy"
  exit 0
else
  echo "✗ $SERVICE is unhealthy"
  echo "$STATUS"
  exit 1
fi

Run it periodically with cron:

*/5 * * * * /path/to/health-check.sh

External monitoring

Use external monitoring services to check your endpoints:

Uptime Robot: Free uptime monitoring
Pingdom: Comprehensive monitoring with alerts
Better Uptime: Modern uptime monitoring
StatusCake: Free and paid monitoring

Resource usage

Uncloud doesn’t provide built-in metrics yet, but you can check resource usage on machines:

System resources

SSH into a machine and check:

# CPU and memory usage
top

# Detailed memory info
free -h

# Disk usage
df -h

# Docker disk usage
docker system df

Container resources

Check resource usage for a specific container:

docker stats web-api-a7f3

For all containers:

docker stats

WireGuard network stats

Check WireGuard interface stats:

sudo wg show

This shows:

Peer connections
Endpoint addresses
Latest handshakes
Data transfer statistics

Why doesn't Uncloud have built-in metrics?

Uncloud focuses on simplicity and minimizing overhead. Built-in metrics and monitoring add significant complexity and resource usage.For production workloads, we recommend:

Exporting logs to external services (e.g., Datadog, Grafana Loki)
Using Prometheus exporters in your services
Setting up external uptime monitoring

This approach keeps Uncloud lean while giving you flexibility to choose the best monitoring tools for your needs.

Monitoring best practices

Set up log aggregation

For production, export logs to a centralized system:

Grafana Loki: Open-source log aggregation
Datadog: Full-stack monitoring platform
Papertrail: Simple log management
Logtail: Modern log management

Ship logs from containers using a sidecar or Docker logging driver.

Monitor critical paths

Focus on monitoring:

Service availability (are containers running?)
HTTPS certificate expiration
Disk space on machines
WireGuard connectivity between machines

Alert on failures

Set up alerts for:

Containers that restart frequently
Machines going offline
TLS certificate renewal failures
Disk space above 85% usage

Regular health checks

Run daily checks:

# Check cluster health
uc machine ls
uc service ls

# Check disk usage on machines
for machine in $(uc machine ls --format '{{.Name}}'); do
  echo "Disk usage on $machine:"
  ssh $machine "df -h | grep -v tmpfs"
done

Log retention

Manage log retention to prevent disk space issues:

# Limit systemd journal size
sudo journalctl --vacuum-size=1G
sudo journalctl --vacuum-time=30d

Debugging commands

Useful commands for troubleshooting:

# Execute commands in a container
uc service exec web-api ps aux
uc service exec web-api cat /etc/resolv.conf

# Check container network connectivity
uc service exec web-api ping api-service.internal
uc service exec web-api curl http://api-service.internal:8000

# Inspect container config
uc service inspect web-api

# Check Caddy configuration
uc caddy config

# View recent events from a service
uc service logs --since 10m web-api

Monitoring command reference

Command	Description
`uc service ls`	List all services
`uc service inspect SERVICE`	Show detailed service info
`uc service logs SERVICE`	View service logs
`uc service logs -f SERVICE`	Follow logs in real-time
`uc service logs -n NUM SERVICE`	Limit log lines per replica
`uc service logs --since TIME SERVICE`	Filter logs by time
`uc service logs -m MACHINE SERVICE`	Filter logs by machine
`uc service exec SERVICE COMMAND`	Execute command in container
`uc machine ls`	List all machines
`uc ps`	List all containers

Get Started

Core Concepts

Deployment

Operations

Advanced

​Service status

​List all services

​Inspect a specific service

​Quick status with uc ps

​Container logs

​View service logs

​Follow logs in real-time

​Limit log lines

​Time-based filtering

​Filter by machine

​View logs for multiple services

​View logs from all services

​UTC timestamps

​System logs

​Uncloud daemon logs

​Corrosion logs

​Docker logs

​System logs

​Health check monitoring

​Container state

​Health check scripts

​External monitoring

​Resource usage

​System resources

​Container resources

​WireGuard network stats

​Monitoring best practices

​Set up log aggregation

​Monitor critical paths

​Alert on failures

​Regular health checks

​Log retention

​Debugging commands

​Monitoring command reference

​Next steps

Troubleshooting

Service Management

Build docs developers (and LLMs) love

Service status

List all services

Inspect a specific service

Quick status with `uc ps`

Container logs

View service logs

Follow logs in real-time

Limit log lines

Time-based filtering

Filter by machine

View logs for multiple services

View logs from all services

UTC timestamps

System logs

Uncloud daemon logs

Corrosion logs

Docker logs

System logs

Health check monitoring

Container state

Health check scripts

External monitoring

Resource usage

System resources

Container resources

WireGuard network stats

Monitoring best practices

Set up log aggregation

Monitor critical paths

Alert on failures

Regular health checks

Log retention

Debugging commands

Monitoring command reference

Next steps