Validator Operations
This guide covers the common operations you’ll perform when running a validator.
Starting and Stopping
Starting the Validator
If running as a systemd service:
sudo systemctl start agave-validator
agave-validator --config ~/validator.conf
Stopping the Validator
Always perform a graceful shutdown to avoid corruption and ensure proper tower state saving.
sudo systemctl stop agave-validator
pkill -SIGTERM agave-validator
sudo journalctl -u agave-validator -f
Restarting the Validator
sudo systemctl restart agave-validator
Monitoring Status
Check Validator Status
View your validator in the gossip network:
solana gossip | grep $(solana-keygen pubkey ~/validator-keypair.json)
solana catchup $(solana-keygen pubkey ~/validator-keypair.json)
Slot: 12345678 (100.00% complete)
Your validator is caught up.
Monitor Vote Account
View your vote account details:
solana vote-account $(solana-keygen pubkey ~/vote-account-keypair.json)
- Current vote credits
- Commission rate
- Last vote slot
- Root slot
- Active stake
Check Block Production
View recent block production:
Filter to your validator:
solana block-production | grep $(solana-keygen pubkey ~/validator-keypair.json)
View Validator Balance
Check your validator identity balance:
solana balance ~/validator-keypair.json
solana balance $(solana-keygen pubkey ~/vote-account-keypair.json)
Health Checks
RPC Health Check
If you’re running RPC, check health via HTTP:
curl -X POST -H "Content-Type: application/json" -d '{
"jsonrpc": "2.0",
"id": 1,
"method": "getHealth"
}' http://localhost:8899
{"jsonrpc":"2.0","result":"ok","id":1}
Admin RPC Commands
The validator provides admin RPC commands for management. These require access to the admin RPC socket.
View your validator’s contact information:
agave-validator contact-info
Exit Validator
Gracefully exit the validator:
With forced exit:
agave-validator exit --force
Monitor Validator
Continuously monitor validator status:
Process Health
Check if the validator process is running:
ps aux | grep agave-validator
top -p $(pgrep agave-validator)
Log Management
View Logs
If using systemd:
sudo journalctl -u agave-validator -f
--log):
Filter Logs
Show only errors:
sudo journalctl -u agave-validator | grep ERROR
sudo journalctl -u agave-validator | grep WARN
sudo journalctl -u agave-validator | grep "vote"
Log Rotation
Systemd automatically handles log rotation. For file-based logs, configure logrotate.
Create /etc/logrotate.d/agave-validator:
/home/sol/validator.log {
daily
rotate 7
compress
delaycompress
missingok
notifempty
copytruncate
}
Change Log Level at Runtime
You can adjust the log level without restarting:
agave-validator set-log-filter <FILTER>
agave-validator set-log-filter infoagave-validator set-log-filter solana_core=debugagave-validator set-log-filter solana_runtime::message_processor=trace,solana_runtime::system_instruction_processor=trace
Check Ledger Size
du -sh ~/validator-ledger
Monitor Disk I/O
Monitor Network Usage
Or use:
Check Memory Usage
Detailed process memory:
pmap -x $(pgrep agave-validator)
Ledger Maintenance
Limit Ledger Size
The --limit-ledger-size flag automatically maintains ledger size:
--limit-ledger-size 50000000
Manual Cleanup
Only perform manual cleanup when the validator is stopped.
sudo systemctl stop agave-validator
rm -rf ~/validator-ledger/rocksdb
sudo systemctl start agave-validator
Snapshot Management
Control snapshot generation with these flags:
--snapshot-interval-slots 500
--full-snapshot-interval-slots 25000
--maximum-full-snapshots-to-retain 5
--maximum-incremental-snapshots-to-retain 10
Disable Snapshots
To disable snapshot generation:
Updating the Validator
Update Process
- Build or download the new version
- Stop the validator
- Replace the binary
- Start the validator
cd ~/salsa
git pull
./cargo build --release
sudo systemctl stop agave-validator
sudo cp target/release/agave-validator /usr/local/bin/
sudo systemctl start agave-validator
Verify Update
Check the version:
agave-validator --version
Vote Account Management
Update Commission
solana vote-update-commission \
$(solana-keygen pubkey ~/vote-account-keypair.json) \
10 \
~/authorized-withdrawer-keypair.json
Withdraw from Vote Account
solana withdraw-from-vote-account \
$(solana-keygen pubkey ~/vote-account-keypair.json) \
<DESTINATION_ADDRESS> \
<AMOUNT> \
~/authorized-withdrawer-keypair.json
Change Vote Authority
solana vote-authorize-voter \
$(solana-keygen pubkey ~/vote-account-keypair.json) \
~/validator-keypair.json \
<NEW_VOTER_PUBKEY>
Delinquency Recovery
If your validator becomes delinquent:
- Check network connectivity: Ensure your validator can reach other nodes
- Verify system resources: Check CPU, memory, and disk I/O
- Review logs: Look for errors or warnings
- Check catchup status:
solana catchup $(solana-keygen pubkey ~/validator-keypair.json)
- Consider a restart: If the issue persists, restart the validator
Emergency Procedures
Force Stop
Only use in emergencies:
sudo killall -9 agave-validator
Reset Tower
Only reset tower state if instructed by the Harmonic team or during coordinated cluster restarts.
sudo systemctl stop agave-validator
rm ~/validator-ledger/tower-*.bin
sudo systemctl start agave-validator
Next Steps
Configuration
Explore advanced configuration options
Monitoring
Set up comprehensive monitoring
Troubleshooting
Troubleshoot common issues
