Overview
The /metrics endpoint exposes Prometheus-compatible metrics in text format, providing comprehensive observability into Permission Mongo’s performance, resource usage, and operational health. This endpoint is designed for integration with Prometheus, Grafana, and other monitoring tools.
Endpoint
Authentication
No authentication required. This endpoint is publicly accessible.
In production environments, consider restricting access to this endpoint using network policies or reverse proxy rules to prevent unauthorized metric collection.
Request
No request parameters or body required.
Example Request
curl http://localhost:8080/metrics
Response
Success Response (200 OK)
Returns Prometheus text format metrics.
# HELP permission_mongo_http_requests_total Total number of HTTP requests by method, path, and status code
# TYPE permission_mongo_http_requests_total counter
permission_mongo_http_requests_total{method="GET",path="/users",status="200"} 1523
permission_mongo_http_requests_total{method="POST",path="/users",status="201"} 342
# HELP permission_mongo_http_request_duration_seconds HTTP request latency in seconds
# TYPE permission_mongo_http_request_duration_seconds histogram
permission_mongo_http_request_duration_seconds_bucket{method="GET",path="/users",le="0.001"} 1205
permission_mongo_http_request_duration_seconds_bucket{method="GET",path="/users",le="0.005"} 1480
permission_mongo_http_request_duration_seconds_bucket{method="GET",path="/users",le="0.01"} 1502
permission_mongo_http_request_duration_seconds_bucket{method="GET",path="/users",le="+Inf"} 1523
permission_mongo_http_request_duration_seconds_sum{method="GET",path="/users"} 2.145
permission_mongo_http_request_duration_seconds_count{method="GET",path="/users"} 1523
# HELP permission_mongo_server_goroutines Number of active goroutines
# TYPE permission_mongo_server_goroutines gauge
permission_mongo_server_goroutines 42
# HELP permission_mongo_mongo_operations_total Total number of MongoDB operations by collection and operation
# TYPE permission_mongo_mongo_operations_total counter
permission_mongo_mongo_operations_total{collection="users",operation="find"} 1523
permission_mongo_mongo_operations_total{collection="users",operation="insert"} 342
# HELP permission_mongo_cache_hits_total Total number of cache hits by cache type
# TYPE permission_mongo_cache_hits_total counter
permission_mongo_cache_hits_total{type="query"} 856
Available Metrics
HTTP Metrics
| Metric Name | Type | Labels | Description |
|---|
permission_mongo_http_requests_total | Counter | method, path, status | Total HTTP requests by method, path, and status code |
permission_mongo_http_request_duration_seconds | Histogram | method, path | HTTP request latency in seconds |
permission_mongo_http_request_size_bytes | Histogram | method, path | HTTP request body size in bytes |
permission_mongo_http_response_size_bytes | Histogram | method, path | HTTP response body size in bytes |
permission_mongo_http_active_requests | Gauge | | Number of currently active HTTP requests |
MongoDB Metrics
| Metric Name | Type | Labels | Description |
|---|
permission_mongo_mongo_operations_total | Counter | collection, operation | Total MongoDB operations by collection and operation type |
permission_mongo_mongo_operation_duration_seconds | Histogram | collection, operation | MongoDB operation latency in seconds |
permission_mongo_mongo_errors_total | Counter | collection, operation | Total MongoDB errors by collection and operation |
permission_mongo_mongo_pool_size | Gauge | | Current MongoDB connection pool size |
Cache Metrics (Redis)
| Metric Name | Type | Labels | Description |
|---|
permission_mongo_cache_hits_total | Counter | type | Total cache hits by cache type |
permission_mongo_cache_misses_total | Counter | type | Total cache misses by cache type |
permission_mongo_cache_operation_duration_seconds | Histogram | operation | Cache operation latency in seconds |
permission_mongo_redis_pool_size | Gauge | | Current Redis connection pool size |
permission_mongo_redis_pool_idle_connections | Gauge | | Current Redis idle connection count |
RBAC Metrics
| Metric Name | Type | Labels | Description |
|---|
permission_mongo_rbac_evaluations_total | Counter | action, result | Total RBAC evaluations by action and result (allowed/denied) |
permission_mongo_rbac_evaluation_duration_seconds | Histogram | | RBAC policy evaluation latency in seconds |
permission_mongo_rbac_cache_size | Gauge | | Number of cached RBAC AST expressions |
Audit Metrics
| Metric Name | Type | Labels | Description |
|---|
permission_mongo_audit_logs_total | Counter | action, success | Total audit log entries by action and success status |
permission_mongo_audit_logs_dropped_total | Counter | | Total audit logs dropped due to full buffer |
permission_mongo_audit_queue_size | Gauge | | Current audit log queue depth |
permission_mongo_audit_batch_size | Histogram | | Size of audit log batches written to MongoDB |
Server Metrics
| Metric Name | Type | Labels | Description |
|---|
permission_mongo_server_info | Gauge | version, go_version | Server build and version information |
permission_mongo_server_uptime_seconds | Gauge | | Server uptime in seconds |
permission_mongo_server_goroutines | Gauge | | Number of active goroutines |
Metric Types
Counter
Monotonically increasing values that represent cumulative totals:
- Request counts
- Error counts
- Operation counts
Gauge
Values that can increase or decrease:
- Active connections
- Goroutine count
- Queue sizes
Histogram
Distributions of values with configurable buckets:
- Request duration
- Request/response sizes
- Batch sizes
Buckets for duration metrics:
[0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10] seconds
Buckets for size metrics:
[100, 1000, 10000, 100000, 1000000, 10000000, 100000000] bytes (100B to 100MB)
Implementation Details
The metrics endpoint:
- Updates runtime metrics (goroutine count) before serving
- Uses the standard Prometheus HTTP handler via
promhttp.Handler()
- Adapted to fasthttp using
fasthttpadaptor.NewFastHTTPHandler()
- Bypassed by authentication middleware for fast metric collection
- Returns content in Prometheus text exposition format
Source Code Reference
Implementation: ~/workspace/source/pkg/api/server.go:458
func (s *Server) metricsHandler(ctx *fasthttp.RequestCtx) {
// Update runtime metrics before serving
metrics.GoroutinesCount.Set(float64(runtime.NumGoroutine()))
// Use fasthttpadaptor to serve promhttp handler
handler := fasthttpadaptor.NewFastHTTPHandler(promhttp.Handler())
handler(ctx)
}
~/workspace/source/pkg/metrics/metrics.go
Prometheus Configuration
Add this scrape configuration to your prometheus.yml:
scrape_configs:
- job_name: 'permission-mongo'
scrape_interval: 15s
static_configs:
- targets: ['localhost:8080']
metrics_path: /metrics
Kubernetes Service Monitor
For Kubernetes environments using Prometheus Operator:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: permission-mongo
namespace: monitoring
spec:
selector:
matchLabels:
app: permission-mongo
endpoints:
- port: http
path: /metrics
interval: 15s
Grafana Dashboards
Use these PromQL queries for common dashboards:
Request Rate
rate(permission_mongo_http_requests_total[5m])
Error Rate
rate(permission_mongo_http_requests_total{status=~"5.."}[5m])
Request Latency (p95)
histogram_quantile(0.95, rate(permission_mongo_http_request_duration_seconds_bucket[5m]))
Cache Hit Rate
sum(rate(permission_mongo_cache_hits_total[5m])) /
(sum(rate(permission_mongo_cache_hits_total[5m])) + sum(rate(permission_mongo_cache_misses_total[5m])))
Use Cases
- Performance Monitoring: Track request latency and throughput
- Capacity Planning: Monitor resource usage and connection pools
- Alerting: Set up alerts for error rates and latency thresholds
- Debugging: Correlate metrics with application behavior
- SLO Tracking: Measure service level objectives
Best Practices
- Scrape Interval: Use 15-30 second intervals for balanced resolution and load
- Cardinality Control: Be mindful of high-cardinality labels (e.g., document IDs)
- Retention Policy: Configure appropriate retention based on storage capacity
- Alerting: Set up alerts for critical metrics (error rates, latency, downtime)
- Dashboards: Create role-specific dashboards (SRE, developers, business)