Skip to main content

​
Introduction

Permission Mongo includes a comprehensive monitoring stack built on Prometheus and Grafana, providing real-time visibility into system performance, health, and behavior.

​
Architecture

The monitoring infrastructure follows a standard observability pattern: 
┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│  Permission     │────▶│   Prometheus    │────▶│    Grafana      │
│  Mongo App      │     │   (scrapes)     │     │  (visualizes)   │
│  :8080/metrics  │     │   :9090         │     │   :3000         │
└─────────────────┘     └─────────────────┘     └─────────────────┘

​
Components

Application

Exposes Prometheus metrics at /metrics endpoint on port 8080

Prometheus

Scrapes metrics every 5 seconds and stores time-series data

Grafana

Provides visualization dashboards and alerting capabilities

​
Quick Start

1

Start the monitoring stack

docker-compose -f docker-compose.monitoring.yaml up -d
2

Access the services

ServiceURLCredentials
Prometheushttp://localhost:9090-
Grafanahttp://localhost:3000admin / admin
Application Metricshttp://localhost:8080/metrics-
3

View the dashboard

The pre-built Permission Mongo dashboard is automatically provisioned in Grafana

​
Metrics Categories

Permission Mongo exposes metrics across six key areas:

​
HTTP Metrics

Track request rates, latency, error rates, and active connections

​
MongoDB Metrics

Monitor database operations, connection pools, and query performance

​
Cache Metrics

Measure cache hit rates, operation latency, and effectiveness

​
RBAC Metrics

Track policy evaluations, authorization decisions, and evaluation performance

​
Audit Metrics

Monitor audit log queue depth, drop rates, and batch sizes

​
Server Metrics

Track server health including uptime, goroutines, and build information

​
Configuration

All monitoring configuration files are located in the monitoring/ directory: 
monitoring/
├── prometheus.yml          # Prometheus scrape configuration
├── alerts.yml              # Alerting rules
├── grafana/
│   ├── provisioning/
│   │   ├── datasources/
│   │   │   └── prometheus.yml  # Auto-configure Prometheus datasource
│   │   └── dashboards/
│   │       └── default.yml     # Dashboard provisioning config
│   └── dashboards/
│       └── permission-mongo.json  # Pre-built dashboard
└── README.md

​
Scrape Interval

The default scrape interval is 5 seconds for the application and 15 seconds globally. This provides detailed granularity for performance analysis.
prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'permission-mongo'
    static_configs:
      - targets: ['permission-mongo:8080']
    scrape_interval: 5s

​
Running Without Docker

If you need to run components separately: 
docker run -d -p 9090:9090 \
  -v $(pwd)/monitoring/prometheus.yml:/etc/prometheus/prometheus.yml \
  prom/prometheus

docker run -d -p 3000:3000 grafana/grafana
Then manually configure:

​
Troubleshooting

  • Check target status: http://localhost:9090/targets
  • Verify application is running and /metrics endpoint is accessible
  • Check network connectivity between containers
  • Verify Prometheus datasource is configured correctly
  • Check metric names match between code and dashboard queries
  • Ensure sufficient time has passed for data collection
  • Path normalization is applied in MetricsMiddleware to prevent cardinality explosion
  • Avoid adding high-cardinality labels (user IDs, request IDs, etc.)

​
Next Steps

Explore Metrics

View all available Prometheus metrics and example queries

Setup Dashboards

Learn about pre-built dashboards and create custom panels

Configure Alerts

Set up alerting rules and notification channels

Build docs developers (and LLMs) love

Get started for freeTalk to us