Troubleshooting Guide

This guide covers common issues encountered when running Sui nodes and their solutions.

Node Won’t Start

Configuration File Errors

Symptoms: Node fails to start with configuration parse errors

# Check logs
journalctl -u sui-node -n 50

# Common error:
# "Failed to load config: invalid type: string, expected a sequence"

Solutions:

Validate YAML syntax

# Install yamllint if needed
sudo apt install yamllint

# Check syntax
yamllint /opt/sui/config/validator.yaml

Check indentation

YAML is whitespace-sensitive. Ensure proper indentation:

# Correct
p2p-config:
  listen-address: 0.0.0.0:8084

# Incorrect (wrong indentation)
p2p-config:
 listen-address: 0.0.0.0:8084

Verify paths

# Check that all file paths exist
ls -l /opt/sui/key-pairs/*.key
ls -l /opt/sui/config/genesis.blob

Permission Issues

Symptoms: “Permission denied” errors

# Error in logs:
# "Failed to open database: Permission denied"

Solution:

# Fix ownership
sudo chown -R sui:sui /opt/sui/db
sudo chown -R sui:sui /opt/sui/config
sudo chown -R sui:sui /opt/sui/key-pairs

# Fix permissions
sudo chmod 600 /opt/sui/key-pairs/*.key
sudo chmod 644 /opt/sui/config/validator.yaml
sudo chmod 644 /opt/sui/config/genesis.blob

Missing Genesis File

Symptoms: “Genesis file not found” Solution:

# Download genesis.blob
cd /opt/sui/config

# For mainnet
sudo wget https://github.com/MystenLabs/sui-genesis/raw/main/mainnet/genesis.blob

# For testnet
sudo wget https://github.com/MystenLabs/sui-genesis/raw/main/testnet/genesis.blob

sudo chown sui:sui genesis.blob

Port Already in Use

Symptoms: “Address already in use”

# Check which process is using the port
sudo lsof -i :8080
sudo lsof -i :9184

# Or using netstat
sudo netstat -tulpn | grep :8080

Solutions:

# Option 1: Kill the process using the port
sudo kill <PID>

# Option 2: Change port in configuration
# Edit validator.yaml and change the conflicting port

# Option 3: Stop conflicting service
sudo systemctl stop <service-name>

Sync Issues

Node Not Syncing

Symptoms: Checkpoint number not increasing

# Check current checkpoint
curl -s http://localhost:9184/metrics | grep highest_synced_checkpoint

# Check again after 1 minute - should increase

Diagnostic Steps:

Check peer connections

curl -s http://localhost:9184/metrics | grep connected_peers

# Should show > 0 peers
# If 0, check network connectivity

Check network connectivity

# Test external connectivity
curl -v https://checkpoints.mainnet.sui.io

# Check firewall rules
sudo iptables -L -n
sudo ufw status

Check state sync configuration

Verify state-archive-read-config in your config:

state-archive-read-config:
  - ingestion-url: https://checkpoints.mainnet.sui.io
    concurrency: 5

Check logs for sync errors

journalctl -u sui-node -f | grep -i "state_sync\|checkpoint"

Slow Sync Speed

Symptoms: Syncing but very slowly Solutions:

# Increase checkpoint download concurrency
# In validator.yaml:
p2p-config:
  state-sync:
    checkpoint-header-download-concurrency: 400
    checkpoint-content-download-concurrency: 400
    checkpoint-content-download-tx-concurrency: 50000

# Check network bandwidth
iftop  # Install: sudo apt install iftop

# Check disk I/O
iotop  # Install: sudo apt install iotop

# Verify using fast storage (NVMe SSD)
lsblk

Stuck at Specific Checkpoint

Symptoms: Sync stops at a specific checkpoint number Solution:

# Check logs for specific errors
journalctl -u sui-node --since "10 minutes ago" | grep -i error

# If database corruption suspected:
# 1. Stop node
sudo systemctl stop sui-node

# 2. Backup current state
sudo mv /opt/sui/db /opt/sui/db.backup

# 3. Restore from recent snapshot or resync
# See snapshots documentation

# 4. Restart
sudo systemctl start sui-node

Performance Issues

High CPU Usage

Symptoms: CPU usage consistently > 80% Diagnostic:

# Check CPU usage
top -u sui

# Check which threads are busy
top -H -p $(pgrep sui-node)

Solutions:

# Reduce checkpoint execution concurrency
# In validator.yaml:
checkpoint-executor-config:
  checkpoint-execution-max-concurrency: 100  # Reduce from 200

# Reduce consensus pending transactions (validators)
consensus-config:
  max-pending-transactions: 10000  # Reduce from 20000

High Memory Usage

Symptoms: Memory usage approaching system limits Diagnostic:

# Check memory usage
free -h

# Check process memory
sudo ps aux | grep sui-node

# Check for memory leaks
sudo pmap $(pgrep sui-node) | tail -1

Solutions:

# Reduce cache sizes in validator.yaml:
execution-cache:
  writeback-cache:
    max-cache-size: 50000  # Reduce from 100000
    object-cache-size: 50000
    transaction-cache-size: 50000

# Or use environment variables:
export SUI_MAX_CACHE_SIZE=50000

Disk I/O Bottleneck

Symptoms: High disk wait times, slow checkpoint execution Diagnostic:

# Check I/O wait
top  # Look at 'wa' percentage

# Detailed I/O stats
iostat -x 5  # Install: sudo apt install sysstat

# Per-process I/O
sudo iotop

Solutions:

Upgrade to NVMe SSD if using SATA SSD
Use dedicated disk for database
Enable aggressive pruning:

authority-store-pruning-config:
  num-epochs-to-retain: 0

Network Issues

No Peer Connections

Symptoms: connected_peers metric shows 0 Diagnostic:

# Check if ports are open
sudo netstat -tulpn | grep sui-node

# Check firewall
sudo iptables -L -n
sudo ufw status

# Test port connectivity from external machine
nc -zv <your-ip> 8080
nc -zuv <your-ip> 8084

Solutions:

Open required ports

# UFW
sudo ufw allow 8080/tcp
sudo ufw allow 8084/udp

# iptables
sudo iptables -A INPUT -p tcp --dport 8080 -j ACCEPT
sudo iptables -A INPUT -p udp --dport 8084 -j ACCEPT
sudo iptables-save | sudo tee /etc/iptables/rules.v4

Configure external address

Update validator.yaml:

p2p-config:
  external-address: /dns/your-domain.com/udp/8084
  # Or with IP:
  external-address: /ip4/YOUR.PUBLIC.IP/udp/8084

Check cloud firewall

For cloud providers, ensure security groups allow the ports:

AWS: Check Security Groups
GCP: Check Firewall Rules
Azure: Check Network Security Groups

Connection Timeouts

Symptoms: Frequent timeout errors in logs Solution:

# Increase timeouts in validator.yaml:
p2p-config:
  state-sync:
    timeout-ms: 30000  # Increase from 10000
    checkpoint-content-timeout-ms: 120000  # Increase from 60000

Validator-Specific Issues

Not Producing Blocks

Symptoms: Validator is active but not proposing/voting Diagnostic:

# Check consensus metrics
curl -s http://localhost:9184/metrics | grep -E "current_round|committed_subdags"

# Check validator status on-chain
sui validator display-metadata <your-address>

Solutions:

Ensure validator is in active set
Check all consensus ports are accessible
Verify protocol key is correct
Check for slashing/reporting

Key Mismatch Errors

Symptoms: “Invalid signature” or “Key mismatch” errors Solution:

# Verify keys match on-chain registration
sui validator display-metadata

# If keys changed, update on-chain:
sui validator update-metadata \
  --protocol-pubkey <new-bls-pubkey> \
  --network-pubkey <new-ed25519-pubkey>

# Wait for next epoch for changes to take effect

Database Issues

Database Corruption

Symptoms: “Corruption” errors in logs, node crashes Solution:

Stop the node

sudo systemctl stop sui-node

Backup corrupted database

sudo mv /opt/sui/db /opt/sui/db.corrupted

Restore from snapshot or resync

Option 1: Restore from backup snapshot

sudo tar -xzf /backups/sui-db-latest.tar.gz -C /opt/sui/db
sudo chown -R sui:sui /opt/sui/db

Option 2: Resync from network (slower)

sudo mkdir -p /opt/sui/db
sudo chown -R sui:sui /opt/sui/db

Restart node

sudo systemctl start sui-node
journalctl -u sui-node -f

Disk Full

Symptoms: “No space left on device” errors Immediate Actions:

# Check disk usage
df -h
du -sh /opt/sui/db/*

# Stop node to prevent further writes
sudo systemctl stop sui-node

# Emergency cleanup - reduce retention
# Edit validator.yaml:
authority-store-pruning-config:
  num-epochs-to-retain: 0  # Aggressive pruning
  num-epochs-to-retain-for-checkpoints: 2

# Restart to trigger pruning
sudo systemctl start sui-node

# Monitor disk usage
watch df -h

Long-term Solutions:

Add more storage
Enable aggressive pruning
Increase pruning frequency:

authority-store-pruning-config:
  pruning-run-delay-seconds: 30  # Run more frequently

Docker-Specific Issues

Container Keeps Restarting

Diagnostic:

# Check container status
docker compose ps

# Check container logs
docker compose logs validator

# Check exit code
docker inspect validator | grep -A 5 State

Solutions:

# Fix common issues:
# 1. Check volume mounts
docker compose config

# 2. Check image exists
docker images | grep sui-node

# 3. Rebuild with latest image
docker compose pull
docker compose up -d

Volume Permission Issues

Symptoms: Permission denied errors in Docker logs Solution:

# Find volume location
docker volume inspect <volume-name>

# Fix permissions
sudo chown -R 1000:1000 /var/lib/docker/volumes/<volume-name>/_data

# Or in docker-compose.yaml:
services:
  validator:
    user: "1000:1000"

Debugging Tools

Admin Interface

The admin interface provides runtime debugging:

# Check node info
curl localhost:1337/node-info

# Change log level
curl localhost:1337/logging -d "debug"

# Enable trace logging for specific module
curl localhost:1337/logging -d "info,state_sync=trace"

Metrics Analysis

# Export all metrics for analysis
curl -s http://localhost:9184/metrics > metrics.txt

# Search for errors
grep -i error metrics.txt

# Check specific metrics
grep "checkpoint" metrics.txt | sort

Enable Verbose Logging

Temporarily enable debug logging:

# Via admin interface (temporary)
curl localhost:1337/logging -d "debug,narwhal=trace"

# Via systemd override (permanent)
sudo systemctl edit sui-node

# Add:
[Service]
Environment="RUST_LOG=debug,sui_core=trace,consensus=trace"

# Reload and restart
sudo systemctl daemon-reload
sudo systemctl restart sui-node

Collecting Debug Information

When seeking help, collect:

#!/bin/bash
# collect-debug-info.sh

OUTPUT="debug-$(date +%Y%m%d-%H%M%S)"
mkdir -p $OUTPUT

# System info
uname -a > $OUTPUT/system-info.txt
df -h > $OUTPUT/disk-usage.txt
free -h > $OUTPUT/memory.txt

# Node info
sui-node --version > $OUTPUT/version.txt 2>&1
cp /opt/sui/config/validator.yaml $OUTPUT/ 2>/dev/null

# Metrics
curl -s http://localhost:9184/metrics > $OUTPUT/metrics.txt

# Recent logs
journalctl -u sui-node --since "1 hour ago" > $OUTPUT/logs.txt

# Network
ss -tulpn | grep sui > $OUTPUT/network.txt

# Compress
tar -czf $OUTPUT.tar.gz $OUTPUT
rm -rf $OUTPUT

echo "Debug info collected: $OUTPUT.tar.gz"

Getting Help

Community Support

Discord: Sui Network Discord - #node-operator-support
Forum: Sui Forum
GitHub: Sui Issues

Reporting Issues

When reporting issues, include:

Node type (validator/fullnode)
Network (mainnet/testnet)
Version (sui-node --version)
Configuration (with sensitive data removed)
Error messages from logs
Steps to reproduce
Debug information bundle

Emergency Contacts

For critical validator issues:

Validator Discord channels
Emergency validator contact methods (provided during onboarding)

Running Nodes

Operations

Validator Guide

Troubleshooting Guide

Node Won’t Start

Configuration File Errors

Permission Issues

Missing Genesis File

Port Already in Use

Sync Issues

Node Not Syncing

Slow Sync Speed

Stuck at Specific Checkpoint

Performance Issues

High CPU Usage

High Memory Usage

Disk I/O Bottleneck

Network Issues

No Peer Connections

Connection Timeouts

Validator-Specific Issues

Not Producing Blocks

Key Mismatch Errors

Database Issues

Database Corruption

Disk Full

Docker-Specific Issues

Container Keeps Restarting

Volume Permission Issues

Debugging Tools

Admin Interface

Metrics Analysis

Enable Verbose Logging

Collecting Debug Information

Getting Help

Community Support

Reporting Issues

Emergency Contacts

Build docs developers (and LLMs) love

Running Nodes

Operations

Validator Guide

​Node Won’t Start

​Configuration File Errors

​Permission Issues

​Missing Genesis File

​Port Already in Use

​Sync Issues

​Node Not Syncing

​Slow Sync Speed

​Stuck at Specific Checkpoint

​Performance Issues

​High CPU Usage

​High Memory Usage

​Disk I/O Bottleneck

​Network Issues

​No Peer Connections

​Connection Timeouts

​Validator-Specific Issues

​Not Producing Blocks

​Key Mismatch Errors

​Database Issues

​Database Corruption

​Disk Full

​Docker-Specific Issues

​Container Keeps Restarting

​Volume Permission Issues

​Debugging Tools

​Admin Interface

​Metrics Analysis

​Enable Verbose Logging

​Collecting Debug Information

​Getting Help

​Community Support

​Reporting Issues

​Emergency Contacts

Build docs developers (and LLMs) love

Node Won’t Start

Configuration File Errors

Permission Issues

Missing Genesis File

Port Already in Use

Sync Issues

Node Not Syncing

Slow Sync Speed

Stuck at Specific Checkpoint

Performance Issues

High CPU Usage

High Memory Usage

Disk I/O Bottleneck

Network Issues

No Peer Connections

Connection Timeouts

Validator-Specific Issues

Not Producing Blocks

Key Mismatch Errors

Database Issues

Database Corruption

Disk Full

Docker-Specific Issues

Container Keeps Restarting

Volume Permission Issues

Debugging Tools

Admin Interface

Metrics Analysis

Enable Verbose Logging

Collecting Debug Information

Getting Help

Community Support

Reporting Issues

Emergency Contacts