Troubleshooting Guide

Overview

This guide covers common NMIS issues, diagnostic commands, and resolution procedures for system administrators.

Diagnostic Tools

Check System Status

# Check all NMIS daemons
/usr/local/nmis9/bin/nmis-cli act=daemon-status

# Check specific services
sudo systemctl status nmis9d
sudo systemctl status mongod

Debug Mode

Enable debug logging for detailed troubleshooting:

# Run command with debug output
/usr/local/nmis9/bin/nmis-cli act=<command> debug=9

# Node admin with debug
/usr/local/nmis9/admin/node_admin.pl act=show node=router1 debug=9

Log Files

Key log file locations:

# Main NMIS log
tail -f /usr/local/nmis9/logs/nmis.log

# CLI operations log
tail -f /usr/local/nmis9/logs/cli.log

# MongoDB log
tail -f /var/log/mongodb/mongod.log

# System logs
journalctl -u nmis9d -f
journalctl -u mongod -f

Common Issues

NMIS Daemons Not Running

Symptoms

No data collection occurring
Web interface shows “no daemons running”
act=daemon-status shows no processes

Diagnosis

Check daemon status

sudo systemctl status nmis9d
/usr/local/nmis9/bin/nmis-cli act=daemon-status

Check for errors in logs

tail -100 /usr/local/nmis9/logs/nmis.log
journalctl -u nmis9d -n 100

Check MongoDB connectivity

sudo systemctl status mongod
mongo --eval "db.serverStatus()"

Resolution

# Start MongoDB if stopped
sudo systemctl start mongod

# Start NMIS daemons
sudo systemctl start nmis9d

# Enable auto-start on boot
sudo systemctl enable nmis9d mongod

# Verify services are running
/usr/local/nmis9/bin/nmis-cli act=daemon-status

Database Connection Failures

Symptoms

“Cannot connect to database” errors
Web interface won’t load
CLI commands fail with database errors

Diagnosis

Check MongoDB status

sudo systemctl status mongod
ps aux | grep mongod

Test database connection

mongo --eval "db.serverStatus()"

Check authentication

# Get credentials from config
grep -E 'db_(username|password|server|port|name)' /usr/local/nmis9/conf/Config.nmis

# Test with credentials
mongo nmisng -u nmisng -p --authenticationDatabase nmisng

Check MongoDB logs

tail -100 /var/log/mongodb/mongod.log

Common Causes

MongoDB not running:

sudo systemctl start mongod
sudo systemctl enable mongod

Authentication failure:

# Re-run database setup
sudo /usr/local/nmis9/admin/setup_mongodb.pl

Connection timeout:

Increase timeout in Config.nmis:

'db_connection_timeout' => 10000,
'db_query_timeout' => 10000,

Firewall blocking connection:

# Allow MongoDB port
sudo firewall-cmd --permanent --add-port=27017/tcp
sudo firewall-cmd --reload

SNMP Collection Failures

Symptoms

Nodes show “Node Down” events
No data being collected from devices
SNMP timeout errors in logs

Diagnosis

Check node configuration

/usr/local/nmis9/admin/node_admin.pl act=show node=<nodename>

Verify:

host - correct IP/hostname
community - SNMP community string
version - SNMP version (snmpv1/snmpv2c/snmpv3)
port - SNMP port (default: 161)

Test SNMP manually

# SNMPv2c
snmpwalk -v 2c -c <community> <host> system

# SNMPv3
snmpwalk -v 3 -u <username> -l authPriv \
  -a SHA -A <authpass> -x AES -X <privpass> \
  <host> system

Check SNMP reachability

# Ping device
ping -c 3 <host>

# Check SNMP port
nc -zv <host> 161

Check NMIS logs

grep <nodename> /usr/local/nmis9/logs/nmis.log | tail -50

Resolution

Incorrect SNMP community:

/usr/local/nmis9/admin/node_admin.pl act=set node=<nodename> \
  entry.configuration.community=<correct-community>

SNMP timeout too short:

/usr/local/nmis9/admin/node_admin.pl act=set node=<nodename> \
  entry.configuration.snmpTimeout=5000

Network connectivity:

Check firewall rules
Verify device SNMP configuration
Check for network issues

SNMPv3 authentication:

/usr/local/nmis9/admin/node_admin.pl act=set node=<nodename> \
  entry.configuration.username=<snmpuser> \
  entry.configuration.authkey=<authpass> \
  entry.configuration.authprotocol=sha \
  entry.configuration.privkey=<privpass> \
  entry.configuration.privprotocol=aes

Polling Failures

Symptoms

Specific nodes not being polled
Irregular polling intervals
Jobs stuck in queue

Diagnosis

# Check scheduled jobs
/usr/local/nmis9/bin/nmis-cli act=list-schedules verbose=true

# Check for stuck jobs
/usr/local/nmis9/bin/nmis-cli act=list-schedules only=active

# Check node status
/usr/local/nmis9/admin/node_admin.pl act=show node=<nodename>

Resolution

Jobs stuck in queue:

# Abort stuck job
/usr/local/nmis9/bin/nmis-cli act=abort id=<job-id>

# Clear all stuck jobs (use with caution)
/usr/local/nmis9/bin/nmis-cli act=delete-schedule id=ALL

Node not being polled:

# Verify node is active
/usr/local/nmis9/admin/node_admin.pl act=set node=<nodename> \
  entry.configuration.collect=true \
  entry.activated.NMIS=true

# Manually schedule collection
/usr/local/nmis9/bin/nmis-cli act=schedule job.type=collect \
  job.node=<nodename>

Restart polling:

sudo systemctl restart nmis9d

Web Interface Issues

Symptoms

Web interface not loading
“403 Forbidden” errors
Graphs not displaying

Diagnosis

Check web server

# Apache
sudo systemctl status httpd
tail -f /var/log/httpd/error_log

# Nginx
sudo systemctl status nginx
tail -f /var/log/nginx/error.log

Check NMIS permissions

ls -la /usr/local/nmis9/htdocs/
ls -la /usr/local/nmis9/cgi-bin/

Check SELinux (if enabled)

sudo getenforce
sudo ausearch -m avc -ts recent

Resolution

Fix file permissions:

sudo /usr/local/nmis9/bin/nmis-cli act=fixperms

Restart web server:

# Apache
sudo systemctl restart httpd

# Nginx  
sudo systemctl restart nginx

SELinux issues:

# Allow httpd to execute NMIS scripts
sudo setsebool -P httpd_can_network_connect 1
sudo setsebool -P httpd_enable_cgi 1

# Or set to permissive mode (not recommended for production)
sudo setenforce 0

Authentication Issues

Symptoms

Cannot login to web interface
“Invalid credentials” errors
User accounts locked

Diagnosis

# Check user sessions
/usr/local/nmis9/bin/nmis-cli act=get-sessions

# Check authentication logs
grep -i "auth" /usr/local/nmis9/logs/nmis.log | tail -50

Resolution

Unlock user account:

/usr/local/nmis9/bin/nmis-cli act=unlock-user user=<username>

Clear user sessions:

/usr/local/nmis9/bin/nmis-cli act=clean-sessions user=<username>

Reset user password: Edit /usr/local/nmis9/conf/Access.nmis or use the web interface (requires admin access).

Performance Issues

Symptoms

Slow web interface
Delayed data collection
High system load

Diagnosis

Check system resources

# CPU and memory
top -b -n 1 | head -20

# Disk usage
df -h

# Disk I/O
iostat -x 2 5

Check MongoDB performance

# Database size
mongo nmisng --eval "db.stats(1024*1024)"

# Current operations
mongo nmisng --eval "db.currentOp()"

# Slow queries
mongo nmisng --eval "db.system.profile.find().sort({ts:-1}).limit(5)"

Check NMIS performance data

/usr/local/nmis9/bin/nmis-cli act=collect-performance-data
/usr/local/nmis9/bin/nmis-cli act=collect-top-data

Resolution

Database cleanup:

# Remove old data
/usr/local/nmis9/bin/nmis-cli act=dbcleanup

# Purge old files
/usr/local/nmis9/bin/nmis-cli act=purge

Optimize database:

# Rebuild indexes
/usr/local/nmis9/bin/nmis-cli act=ensure-indexes drop_unwanted=true

# Compact collections
mongo nmisng --eval "db.events.compact()"
mongo nmisng --eval "db.opstatus.compact()"

Increase system resources:

Add RAM (recommended: 8GB minimum)
Increase disk space
Add CPU cores
Use SSD for database storage

Tune MongoDB: Edit /etc/mongod.conf:

storage:
  wiredTiger:
    engineConfig:
      cacheSizeGB: 2  # Adjust based on available RAM

Disk Space Issues

Symptoms

“No space left on device” errors
Database write failures
Log files not being written

Diagnosis

# Check disk usage
df -h

# Find large directories
du -sh /usr/local/nmis9/* | sort -h
du -sh /var/lib/mongodb/* | sort -h

# Find large files
find /usr/local/nmis9 -type f -size +100M -exec ls -lh {} \;

Resolution

Clean up logs

# Rotate and compress old logs
find /usr/local/nmis9/logs -name "*.log" -mtime +30 -exec gzip {} \;

# Remove very old logs
find /usr/local/nmis9/logs -name "*.log.gz" -mtime +90 -delete

Database cleanup

/usr/local/nmis9/bin/nmis-cli act=dbcleanup info=true
/usr/local/nmis9/bin/nmis-cli act=purge info=true

Remove old RRD files

# Find unused RRD files (for deleted nodes)
find /usr/local/nmis9/database -type f -name "*.rrd" -mtime +90

Clean temporary files

find /tmp -name "nmis*" -mtime +7 -delete
find /usr/local/nmis9/var/nmis_system -name "*.tmp" -delete

Advanced Diagnostics

Database Queries

Useful MongoDB queries for troubleshooting:

// Count nodes by group
db.nodes.aggregate([
  {$group: {_id: "$configuration.group", count: {$sum: 1}}}
])

// Find recent events
db.events.find(
  {active: true},
  {node_uuid: 1, event: 1, startdate: 1}
).sort({startdate: -1}).limit(20)

// Check queue backlog
db.queue.find(
  {in_progress: 0},
  {type: 1, time: 1, priority: 1}
).sort({time: 1}).limit(20)

// Find nodes with errors
db.nodes.find(
  {"configuration.collect": true},
  {name: 1, "configuration.host": 1}
)

Network Diagnostics

# Test SNMP from NMIS server
snmpwalk -v 2c -c public <device-ip> system

# Test connectivity
ping -c 5 <device-ip>
traceroute <device-ip>

# Check for packet loss
mtr --report --report-cycles 100 <device-ip>

# Test SNMP port
nc -zv <device-ip> 161
nmap -sU -p 161 <device-ip>

Process Diagnostics

# Check NMIS processes
ps aux | grep nmis

# Check process tree
pstree -p | grep nmis

# Monitor process resources
top -p $(pgrep -d',' nmis)

# Check open files
lsof -p $(pgrep nmisd.worker)

Emergency Procedures

System Won’t Start

Check critical services:

sudo systemctl status mongod
sudo systemctl status httpd  # or nginx

Check for file system issues:

df -h
sudo dmesg | grep -i error

Review system logs:

journalctl -xe
tail -100 /var/log/messages

Start services manually:

sudo systemctl start mongod
sudo systemctl start nmis9d

Database Corruption

Stop NMIS:

sudo systemctl stop nmis9d

Backup current database:

mongodump --db nmisng --out /tmp/nmis-emergency-backup

Repair database:

mongo nmisng --eval "db.repairDatabase()"

If repair fails, restore from backup:

# See Backup and Restore documentation

Complete System Reset

This procedure will reset NMIS to initial state. Use only as last resort.

# Stop services
sudo systemctl stop nmis9d

# Backup critical data
sudo tar czf /tmp/nmis-emergency-backup.tar.gz \
  /usr/local/nmis9/conf/ \
  /usr/local/nmis9/database/

# Drop and recreate database
mongo nmisng --eval "db.dropDatabase()"

# Re-run setup
sudo /usr/local/nmis9/admin/setup_mongodb.pl

# Fix permissions
sudo /usr/local/nmis9/bin/nmis-cli act=fixperms

# Start services
sudo systemctl start nmis9d

Getting Help

Information to Collect

When requesting support, provide:

System information:

uname -a
cat /etc/os-release
/usr/local/nmis9/bin/nmis-cli --version
mongod --version

Error messages:

tail -100 /usr/local/nmis9/logs/nmis.log
tail -100 /var/log/mongodb/mongod.log
journalctl -u nmis9d -n 100

Configuration:

# Sanitize passwords before sharing
grep -v password /usr/local/nmis9/conf/Config.nmis

System status:

/usr/local/nmis9/bin/nmis-cli act=daemon-status
sudo systemctl status nmis9d mongod httpd
df -h
free -h

Support Resources

Community Forums: https://community.opmantek.com
Documentation: https://community.opmantek.com/display/NMIS
Commercial Support: [email protected]

Getting Started

Core Features

Configuration

Device Models

Administration

Deployment

​Overview

​Diagnostic Tools

​Check System Status

​Debug Mode

​Log Files

​Common Issues

​NMIS Daemons Not Running

​Symptoms

​Diagnosis

​Resolution

​Database Connection Failures

​Symptoms

​Diagnosis

​Common Causes

​SNMP Collection Failures

​Symptoms

​Diagnosis

​Resolution

​Polling Failures

​Symptoms

​Diagnosis

​Resolution

​Web Interface Issues

​Symptoms

​Diagnosis

​Resolution

​Authentication Issues

​Symptoms

​Diagnosis

​Resolution

​Performance Issues

​Symptoms

​Diagnosis

​Resolution

​Disk Space Issues

​Symptoms

​Diagnosis

​Resolution

​Advanced Diagnostics

​Database Queries

​Network Diagnostics

​Process Diagnostics

​Emergency Procedures

​System Won’t Start

​Database Corruption

​Complete System Reset

​Getting Help

​Information to Collect

​Support Resources

​See Also

Build docs developers (and LLMs) love

Overview

Diagnostic Tools

Check System Status

Debug Mode

Log Files

Common Issues

NMIS Daemons Not Running

Symptoms

Diagnosis

Resolution

Database Connection Failures

Symptoms

Diagnosis

Common Causes

SNMP Collection Failures

Symptoms

Diagnosis

Resolution

Polling Failures

Symptoms

Diagnosis

Resolution

Web Interface Issues

Symptoms

Diagnosis

Resolution

Authentication Issues

Symptoms

Diagnosis

Resolution

Performance Issues

Symptoms

Diagnosis

Resolution

Disk Space Issues

Symptoms

Diagnosis

Resolution

Advanced Diagnostics

Database Queries

Network Diagnostics

Process Diagnostics

Emergency Procedures

System Won’t Start

Database Corruption

Complete System Reset

Getting Help

Information to Collect

Support Resources

See Also