Troubleshooting

Overview

This guide covers common issues you may encounter when deploying and operating Ubu-Block nodes, along with step-by-step solutions.

Node Startup Issues

Node Fails to Start

Configuration file not found

Error:

Failed to read config file: No such file or directory

Solution:

# Verify config file exists
ls -la config.toml

# Check file path in command
submission --config /path/to/config.toml

# Use absolute path if needed
submission --config /etc/ubu-block/config.toml

Invalid configuration format

Error:

Failed to parse config file: expected `=`, found `:` at line 3

Solution: TOML uses = not : for key-value pairs:

# Wrong
main_db: "sqlite://./data/blockchain.db"

# Correct
main_db = "sqlite://./data/blockchain.db"

Validate TOML syntax:

# Install taplo
cargo install taplo-cli

# Validate config
taplo check config.toml

Port already in use

Error:

thread 'main' panicked at 'Failed to bind to address: Address already in use'

Solution:

# Find what's using the port
sudo lsof -i :9090
sudo lsof -i :9091

# Option 1: Stop the conflicting process
sudo kill <PID>

# Option 2: Use different ports in config.toml
http_addr = "127.0.0.1:9093"
node_addr = "127.0.0.1:9092"

Database connection errors

Error:

Failed to connect to database: unable to open database file

Solution:

# Check if database directory exists
mkdir -p data

# Check file permissions
ls -la data/

# Ensure correct ownership
sudo chown -R $USER:$USER data/
chmod 700 data
chmod 600 data/*.db

# Verify database path in config
cat config.toml | grep "_db"

Database Issues

Blockchain Validation Failures

Error:

thread 'main' panicked at 'Could not verify block, found 0e70cebe..., block: Block {...}'

Cause: The blockchain has been tampered with or corrupted.

Identify the corrupted block

The error message shows the block hash. Note the block details.

Check recent changes

# Check if someone manually edited the database
sqlite3 data/blockchain.db "SELECT * FROM blocks ORDER BY timestamp DESC LIMIT 5;"

Restore from backup

If data was tampered with:

# Stop the node
sudo systemctl stop ubu-submission

# Restore from latest backup
sqlite3 data/blockchain.db ".restore /backup/ubu-block/latest/blockchain.db"

# Restart node
sudo systemctl start ubu-submission

# Validate
cargo run -- validate

Resync from peers

If backup is unavailable, resync from a trusted peer:

# Remove corrupted database
rm data/blockchain.db

# Start node (will resync from peers)
sudo systemctl start ubu-submission

Database Locked Errors

Error:

database is locked

Solution:

Find Locking Process
Enable WAL Mode
Increase Timeout

# Find processes using the database
sudo fuser data/blockchain.db

# Or using lsof
sudo lsof | grep blockchain.db

# Check for stuck processes
ps aux | grep submission

Write-Ahead Logging allows concurrent access:

# Stop node first
sudo systemctl stop ubu-submission

# Enable WAL mode
sqlite3 data/blockchain.db "PRAGMA journal_mode=WAL;"

# Restart node
sudo systemctl start ubu-submission

# In database connection code, increase busy timeout
sqlite3 data/blockchain.db "PRAGMA busy_timeout = 30000;"

Database Corruption

Symptoms:

PRAGMA integrity_check fails
Random crashes or errors
Inconsistent query results

Recovery Steps:

Attempt database repair

# Dump and restore
sqlite3 data/blockchain.db ".dump" | sqlite3 data/blockchain_repaired.db

# Verify repaired database
sqlite3 data/blockchain_repaired.db "PRAGMA integrity_check;"

# Replace original if successful
mv data/blockchain.db data/blockchain_corrupted.db
mv data/blockchain_repaired.db data/blockchain.db

Restore from backup

If repair fails:

# Find latest good backup
ls -lht /backup/ubu-block/

# Restore
tar -xzf /backup/ubu-block/ubu-block-YYYY-MM-DD.tar.gz -C data/

Network and Connectivity Issues

Cannot Connect to Peers

Network connectivity

# Test basic connectivity
ping -c 4 peer-hostname

# Test specific port
telnet peer-hostname 9090
nc -zv peer-hostname 9090

# Check DNS resolution
nslookup peer-hostname

Firewall blocking connection

# Check firewall status
sudo ufw status

# Allow P2P port
sudo ufw allow 9090/tcp

# Check iptables
sudo iptables -L -n | grep 9090

Incorrect peer address

Verify peer configuration:

config.toml

# Check peer address format
[[peers]]
name = "peer1"
address = "157.230.123.45:9090"  # Must include port

Test peer is listening:

curl http://peer-address:9091/api/v1/health

SSL/TLS certificate issues

If using encrypted connections:

# Test TLS connection
openssl s_client -connect peer-hostname:9090

# Check certificate validity
openssl s_client -connect peer-hostname:9090 -showcerts

API Not Responding

Symptoms:

HTTP requests timeout
Connection refused errors
502/504 gateway errors

Diagnosis:

Check if node is running

sudo systemctl status ubu-submission
ps aux | grep submission

Verify port binding

# Check if listening on correct port
netstat -tuln | grep 9091
ss -tuln | grep 9091

# Check binding address
# Should be 0.0.0.0 for external access, not 127.0.0.1

Check logs for errors

sudo journalctl -u ubu-submission -n 100 --no-pager
sudo journalctl -u ubu-submission -f

Test locally first

# Test from localhost
curl http://127.0.0.1:9091/api/v1/health

# Then test from external IP
curl http://your-server-ip:9091/api/v1/health

Performance Issues

Slow Query Performance

Symptoms:

API requests take too long
Query timeouts
High CPU usage

Solutions:

Optimize Database
Add Indexes
Tune SQLite

# Run ANALYZE to update statistics
sqlite3 data/blockchain.db "ANALYZE;"

# Rebuild indexes
sqlite3 data/blockchain.db ".schema" | grep "CREATE INDEX"

# Vacuum to reclaim space
sqlite3 data/blockchain.db "VACUUM;"

Identify slow queries and add indexes:

-- Check query plan
EXPLAIN QUERY PLAN SELECT * FROM results WHERE station_id = 123;

-- Add index if needed
CREATE INDEX idx_results_station ON results(station_id);

sqlite3 data/blockchain.db <<EOF
PRAGMA cache_size = -64000;  -- 64MB cache
PRAGMA temp_store = MEMORY;
PRAGMA mmap_size = 268435456;  -- 256MB mmap
EOF

High Memory Usage

Diagnosis:

# Check memory usage
ps aux | grep submission
top -p $(pgrep submission)

# Check for memory leaks
valgrind --leak-check=full submission --config config.toml

Solutions:

Reduce database cache size
Limit concurrent connections
Restart node periodically
Upgrade RAM if necessary

High CPU Usage

Common Causes:

Block validation: Normal during validation
Peer synchronization: Temporary during initial sync
Heavy queries: Optimize database queries
Infinite loops: Check logs for errors

# Profile CPU usage
perf top -p $(pgrep submission)

# Check what threads are doing
strace -p $(pgrep submission) -f

Build and Compilation Issues

Rust Edition Not Supported

Error:

error: edition 2024 is unstable and only available with -Z unstable-options

Solution:

# Update Rust to latest stable
rustup update stable

# Verify version
rustc --version  # Should be 1.85.0 or later

# Set as default
rustup default stable

OpenSSL/TLS Build Errors

Error:

failed to run custom build command for `openssl-sys`

Solution:

Ubuntu/Debian
macOS
Use rustls instead

sudo apt update
sudo apt install pkg-config libssl-dev

brew install openssl@3
export OPENSSL_DIR=$(brew --prefix openssl@3)
cargo build --release

Modify Cargo.toml to use rustls:

[dependencies]
sqlx = { version = "0.8", features = [
    "runtime-tokio-rustls",  # Instead of runtime-tokio-native-tls
    "sqlite",
] }

Out of Memory During Build

Error:

error: could not compile `ubu-block` due to out of memory

Solution:

# Limit parallel jobs
cargo build --release -j 1

# Or use 2 jobs
cargo build --release -j 2

# Add swap space (Linux)
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

Data Integrity Issues

Duplicate Blocks

Symptoms:

Validation fails
Duplicate primary key errors

Solution:

# Find duplicates
sqlite3 data/blockchain.db "SELECT hash, COUNT(*) FROM blocks GROUP BY hash HAVING COUNT(*) > 1;"

# Remove duplicates (keep oldest)
sqlite3 data/blockchain.db <<EOF
DELETE FROM blocks
WHERE rowid NOT IN (
    SELECT MIN(rowid)
    FROM blocks
    GROUP BY hash
);
EOF

# Validate
cargo run -- validate

Missing Genesis Block

Error:

Genesis block not found

Solution:

# Reinitialize blockchain
cargo run -- init --source setup_constituencies.sql

# Or restore from backup
tar -xzf /backup/ubu-block/genesis-backup.tar.gz -C data/

Security Issues

Unauthorized Access Attempts

Symptoms:

Unusual log entries
Failed authentication attempts
Unexpected API calls

Actions:

Review logs

# Check for suspicious activity
sudo journalctl -u ubu-submission | grep -i "auth\|fail\|error"

# Check access logs
tail -n 1000 /var/log/nginx/access.log | grep -v "200"

Block malicious IPs

# Block IP with UFW
sudo ufw deny from 192.168.1.100

# Or using iptables
sudo iptables -A INPUT -s 192.168.1.100 -j DROP

Enable rate limiting

Configure nginx or reverse proxy:

limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;

server {
    location /api/ {
        limit_req zone=api burst=20;
        proxy_pass http://127.0.0.1:9091;
    }
}

Getting Help

If you’re still experiencing issues:

GitHub Issues

Report bugs and request features on the GitHub repository

Community Support

Join the community discussions for help from other users

Documentation

Review the full documentation for detailed guides

Logs and Diagnostics

Collect logs and system info before reporting:

sudo journalctl -u ubu-submission > logs.txt
uname -a >> logs.txt
cargo --version >> logs.txt

Diagnostic Checklist

When reporting issues, include:

Ubu-Block version (git rev-parse HEAD)
Operating system and version
Rust version (rustc --version)
Node type (submission, observer, verification)
Configuration file (redact sensitive data)
Full error message and stack trace
Recent log entries
Steps to reproduce the issue
Expected vs actual behavior

Setup

Operations

Overview

Node Startup Issues

Node Fails to Start

Database Issues

Blockchain Validation Failures

Database Locked Errors

Database Corruption

Network and Connectivity Issues

Cannot Connect to Peers

API Not Responding

Performance Issues

Slow Query Performance

High Memory Usage

High CPU Usage

Build and Compilation Issues

Rust Edition Not Supported

OpenSSL/TLS Build Errors

Out of Memory During Build

Data Integrity Issues

Duplicate Blocks

Missing Genesis Block

Security Issues

Unauthorized Access Attempts

Getting Help

GitHub Issues

Community Support

Documentation

Logs and Diagnostics

Diagnostic Checklist

Next Steps

Monitoring

Maintenance

Build docs developers (and LLMs) love

Setup

Operations

​Overview

​Node Startup Issues

​Node Fails to Start

​Database Issues

​Blockchain Validation Failures

​Database Locked Errors

​Database Corruption

​Network and Connectivity Issues

​Cannot Connect to Peers

​API Not Responding

​Performance Issues

​Slow Query Performance

​High Memory Usage

​High CPU Usage

​Build and Compilation Issues

​Rust Edition Not Supported

​OpenSSL/TLS Build Errors

​Out of Memory During Build

​Data Integrity Issues

​Duplicate Blocks

​Missing Genesis Block

​Security Issues

​Unauthorized Access Attempts

​Getting Help

GitHub Issues

Community Support

Documentation

Logs and Diagnostics

​Diagnostic Checklist

​Next Steps

Monitoring

Maintenance

Build docs developers (and LLMs) love

Overview

Node Startup Issues

Node Fails to Start

Database Issues

Blockchain Validation Failures

Database Locked Errors

Database Corruption

Network and Connectivity Issues

Cannot Connect to Peers

API Not Responding

Performance Issues

Slow Query Performance

High Memory Usage

High CPU Usage

Build and Compilation Issues

Rust Edition Not Supported

OpenSSL/TLS Build Errors

Out of Memory During Build

Data Integrity Issues

Duplicate Blocks

Missing Genesis Block

Security Issues

Unauthorized Access Attempts

Getting Help

Diagnostic Checklist

Next Steps