Monitoring - AspNetRun Microservices

This guide covers monitoring, health checks, and observability features in AspNetRun Microservices.

Health Check Endpoints

All microservices implement health checks using ASP.NET Core Health Checks with the HealthChecks.UI.Client for detailed JSON responses.

Service Health Endpoints

Each microservice exposes a /health endpoint:

Service	Health Check URL	Dependencies Checked
Catalog API	http://localhost:6000/health	PostgreSQL (CatalogDb)
Basket API	http://localhost:6001/health	PostgreSQL (BasketDb), Redis
Ordering API	http://localhost:6003/health	SQL Server (OrderDb)
API Gateway	http://localhost:6004/health	None (Gateway health only)

Health Check Response Format

Health checks return detailed JSON responses:

{
  "status": "Healthy",
  "totalDuration": "00:00:00.0123456",
  "entries": {
    "npgsql": {
      "status": "Healthy",
      "duration": "00:00:00.0098765",
      "data": {}
    }
  }
}

Status values:

Healthy - All checks passed
Degraded - Some non-critical checks failed
Unhealthy - Critical checks failed

Checking Health Status

Using curl

# Check Catalog API health
curl http://localhost:6000/health

# Check Basket API health (includes Redis check)
curl http://localhost:6001/health

# Check Ordering API health
curl http://localhost:6003/health

# Check with formatted output
curl http://localhost:6000/health | jq

Using PowerShell

# Check health and parse JSON
Invoke-RestMethod http://localhost:6000/health | ConvertTo-Json

# Check all services
@(6000, 6001, 6003) | ForEach-Object {
    Write-Host "Checking port $_"
    Invoke-RestMethod "http://localhost:$_/health"
}

Health Check Script

Create a monitoring script to check all services:

#!/bin/bash
# check-health.sh

services=(
    "Catalog:6000"
    "Basket:6001"
    "Ordering:6003"
    "Gateway:6004"
)

for service in "${services[@]}"; do
    IFS=':' read -r name port <<< "$service"
    echo "Checking $name API..."
    
    response=$(curl -s http://localhost:$port/health)
    status=$(echo $response | jq -r '.status')
    
    if [ "$status" = "Healthy" ]; then
        echo "✓ $name is healthy"
    else
        echo "✗ $name is $status"
        echo "  Details: $response"
    fi
    echo ""
done

Make it executable and run:

chmod +x check-health.sh
./check-health.sh

Database Health Monitoring

PostgreSQL (Catalog & Basket)

The Catalog and Basket APIs use the AddNpgSql health check:

builder.Services.AddHealthChecks()
    .AddNpgSql(builder.Configuration.GetConnectionString("Database")!);

This verifies:

Database server is reachable
Connection can be established
Authentication is successful

Manual verification:

# Check Catalog database
docker exec -it catalogdb pg_isready -U postgres -d CatalogDb

# Check Basket database
docker exec -it basketdb pg_isready -U postgres -d BasketDb

# Test actual connection
docker exec -it catalogdb psql -U postgres -d CatalogDb -c "SELECT 1;"

Redis (Basket Cache)

The Basket API checks Redis connectivity:

builder.Services.AddHealthChecks()
    .AddNpgSql(builder.Configuration.GetConnectionString("Database")!)
    .AddRedis(builder.Configuration.GetConnectionString("Redis")!);

Manual verification:

# Check Redis connectivity
docker exec -it distributedcache redis-cli ping
# Should return: PONG

# Check Redis info
docker exec -it distributedcache redis-cli info server

# Monitor Redis commands in real-time
docker exec -it distributedcache redis-cli monitor

SQL Server (Ordering)

The Ordering API uses the AddSqlServer health check:

builder.Services.AddHealthChecks()
    .AddSqlServer(configuration.GetConnectionString("Database")!);

Manual verification:

# Check SQL Server is ready
docker exec -it orderdb /opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P SwN12345678 -Q "SELECT @@VERSION"

# Check database exists
docker exec -it orderdb /opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P SwN12345678 -Q "SELECT name FROM sys.databases"

# Test connection to OrderDb
docker exec -it orderdb /opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P SwN12345678 -d OrderDb -Q "SELECT COUNT(*) FROM Orders"

Container Health Monitoring

Docker Container Status

# Check all containers
docker-compose ps

# Check specific service
docker-compose ps catalog.api

# View container health (if health checks are defined in Dockerfile)
docker ps --format "table {{.Names}}\t{{.Status}}"

Resource Usage Monitoring

# Real-time resource usage
docker stats

# Snapshot of current usage
docker stats --no-stream

# Monitor specific containers
docker stats catalog.api basket.api ordering.api

# Format output
docker stats --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}"

Expected resource usage:

Databases: 100-300 MB each
Microservices: 50-150 MB each
RabbitMQ: 100-200 MB
API Gateway: 50-100 MB
Web UI: 50-100 MB

If any service consistently uses >500 MB, investigate for memory leaks.

Application Logging

Log Levels

All services use ASP.NET Core logging with the following default configuration:

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  }
}

Viewing Logs

# View all logs
docker-compose logs

# Follow logs in real-time
docker-compose logs -f

# View specific service logs
docker-compose logs catalog.api
docker-compose logs basket.api
docker-compose logs ordering.api

# Follow multiple services
docker-compose logs -f catalog.api basket.api

# View last N lines
docker-compose logs --tail=100 catalog.api

# View logs since timestamp
docker-compose logs --since 2024-01-01T10:00:00

# View logs with timestamps
docker-compose logs -t catalog.api

Log Patterns to Monitor

Success Patterns

# Successful requests
docker-compose logs catalog.api | grep "HTTP/1.1 200"

# Health check successes
docker-compose logs | grep "health.*Healthy"

# Successful database connections
docker-compose logs | grep -i "database.*connected"

Error Patterns

# Application errors
docker-compose logs | grep -i "error\|exception\|fail"

# Database connection errors
docker-compose logs | grep -i "connection.*failed\|timeout"

# HTTP errors
docker-compose logs | grep "HTTP/1.1 [45][0-9][0-9]"

Structured Logging

Services use structured logging with MediatR behaviors: LoggingBehavior - Logs all MediatR requests/responses:

Request name and data
Execution time
Response status

ValidationBehavior - Logs validation failures:

Validation errors
Invalid request data

Example log output:

[INF] Handling GetProductsQuery
[INF] GetProductsQuery handled in 45ms
[WRN] Validation failed for CreateProductCommand: Name is required

RabbitMQ Monitoring

Management UI

Access the RabbitMQ Management Console:

URL: http://localhost:15672
Username: guest
Password: guest

Key Metrics to Monitor

Queues

Navigate to Queues tab
Look for the BasketCheckout queue
Monitor:
- Ready: Messages waiting to be consumed
- Unacked: Messages being processed
- Total: Total messages in queue

In a healthy system, messages should not accumulate. If “Ready” count keeps growing, the Ordering service may not be consuming messages.

Exchanges

MassTransit creates these exchanges:

BasketCheckout - For basket checkout events
Service-specific exchanges for request/response patterns

Connections

Verify both services are connected:

Go to Connections tab
Should see connections from:
- basket.api (publisher)
- ordering.api (consumer)

CLI Monitoring

# List queues
docker exec -it messagebroker rabbitmqctl list_queues

# List exchanges
docker exec -it messagebroker rabbitmqctl list_exchanges

# List connections
docker exec -it messagebroker rabbitmqctl list_connections

# Check cluster status
docker exec -it messagebroker rabbitmqctl cluster_status

Testing Message Flow

Complete a checkout in the Web UI (https://localhost:6065)
Open RabbitMQ Management UI (http://localhost:15672)
Navigate to Queues → BasketCheckout
You should see:
- Message published by Basket API
- Message consumed by Ordering API
- Message count returns to 0

Check Ordering API logs:

docker-compose logs ordering.api | grep -i "basketcheckout"

Service-Specific Monitoring

Catalog API

What to monitor:

PostgreSQL connection health
Marten document store operations
Product query performance

Key endpoints:

# Health check
curl http://localhost:6000/health

# Test read operation
curl http://localhost:6000/products

# Test single product
curl http://localhost:6000/products/{id}

Logs to watch:

docker-compose logs -f catalog.api | grep -i "marten\|postgres\|product"

Basket API

What to monitor:

PostgreSQL connection (for cart persistence)
Redis connection (for caching)
gRPC calls to Discount service
RabbitMQ message publishing

Key endpoints:

# Health check (includes PostgreSQL + Redis)
curl http://localhost:6001/health

# Test basket retrieval
curl http://localhost:6001/basket/testuser

Logs to watch:

# Cache operations
docker-compose logs -f basket.api | grep -i "cache\|redis"

# gRPC calls
docker-compose logs -f basket.api | grep -i "grpc\|discount"

# Message publishing
docker-compose logs -f basket.api | grep -i "rabbitmq\|masstransit"

Ordering API

What to monitor:

SQL Server connection health
Entity Framework migrations
RabbitMQ message consumption
Order creation from events

Key endpoints:

# Health check
curl http://localhost:6003/health

# Test orders endpoint
curl http://localhost:6003/orders

Logs to watch:

# Database operations
docker-compose logs -f ordering.api | grep -i "sql\|database\|migration"

# Message consumption
docker-compose logs -f ordering.api | grep -i "basketcheckout\|consumer"

# Order processing
docker-compose logs -f ordering.api | grep -i "order"

Discount gRPC Service

What to monitor:

SQLite database operations
gRPC service calls from Basket API

Logs to watch:

docker-compose logs -f discount.grpc | grep -i "discount\|grpc"

Testing gRPC: Use grpcurl (if installed):

grpcurl -plaintext -d '{"productName": "IPhone X"}' localhost:6002 discount.DiscountProtoService/GetDiscount

API Gateway (YARP)

What to monitor:

Request routing
Rate limiting (Ordering service)
Backend service health

Logs to watch:

# Routing decisions
docker-compose logs -f yarpapigateway | grep -i "route\|proxy"

# Rate limiting
docker-compose logs -f yarpapigateway | grep -i "rate.*limit"

# Backend failures
docker-compose logs -f yarpapigateway | grep -i "error\|fail"

Test rate limiting:

# Send multiple requests to ordering service
for i in {1..20}; do
    curl -i http://localhost:6004/ordering-service/orders
    echo "Request $i"
    sleep 0.1
done
# Should see 429 (Too Many Requests) after hitting the limit

Performance Monitoring

Response Time Monitoring

Create a simple script to monitor endpoint response times:

#!/bin/bash
# monitor-performance.sh

while true; do
    echo "[$(date +%T)] Checking response times..."
    
    # Catalog
    time curl -s -o /dev/null http://localhost:6000/products
    
    # Basket
    time curl -s -o /dev/null http://localhost:6001/basket/testuser
    
    # Ordering
    time curl -s -o /dev/null http://localhost:6003/orders
    
    echo ""
    sleep 10
done

Database Query Performance

PostgreSQL:

-- View slow queries
SELECT pid, now() - pg_stat_activity.query_start AS duration, query 
FROM pg_stat_activity 
WHERE state = 'active' 
ORDER BY duration DESC;

-- View database statistics
SELECT * FROM pg_stat_database WHERE datname = 'CatalogDb';

SQL Server:

-- View currently executing queries
SELECT 
    r.session_id,
    r.status,
    r.command,
    r.cpu_time,
    r.total_elapsed_time,
    t.text
FROM sys.dm_exec_requests r
CROSS APPLY sys.dm_exec_sql_text(r.sql_handle) t;

Alerting Considerations

While this project doesn’t include built-in alerting, consider monitoring:

Critical Alerts

Health check failures for >2 minutes
Database connection failures
RabbitMQ queue depth >1000 messages
Container restarts
Container memory usage >80%

Warning Alerts

Response times >1 second
RabbitMQ queue depth >100 messages
Failed validation rate >10%
Container CPU usage >70%

Production Monitoring Recommendations

For production deployments, consider adding:

Application Performance Monitoring (APM)
- Application Insights
- New Relic
- Datadog
Distributed Tracing
- OpenTelemetry
- Jaeger
- Zipkin
Centralized Logging
- ELK Stack (Elasticsearch, Logstash, Kibana)
- Seq
- Grafana Loki
Metrics Collection
- Prometheus
- Grafana
- StatsD
Health Check UI
- AspNetCore.HealthChecks.UI
- Custom dashboard

Next Steps

Running Locally - Setup guide
Troubleshooting - Common issues and solutions
Configuration - Environment variables and settings

Docker

Configuration

Operations

​Health Check Endpoints

​Service Health Endpoints

​Health Check Response Format

​Checking Health Status

​Using curl

​Using PowerShell

​Health Check Script

​Database Health Monitoring

​PostgreSQL (Catalog & Basket)

​Redis (Basket Cache)

​SQL Server (Ordering)

​Container Health Monitoring

​Docker Container Status

​Resource Usage Monitoring

​Application Logging

​Log Levels

​Viewing Logs

​Log Patterns to Monitor

​Success Patterns

​Error Patterns

​Structured Logging

​RabbitMQ Monitoring

​Management UI

​Key Metrics to Monitor

​Queues

​Exchanges

​Connections

​CLI Monitoring

​Testing Message Flow

​Service-Specific Monitoring

​Catalog API

​Basket API

​Ordering API

​Discount gRPC Service

​API Gateway (YARP)

​Performance Monitoring

​Response Time Monitoring

​Database Query Performance

​Alerting Considerations

​Critical Alerts

​Warning Alerts

​Production Monitoring Recommendations

​Next Steps

Build docs developers (and LLMs) love

Health Check Endpoints

Service Health Endpoints

Health Check Response Format

Checking Health Status

Using curl

Using PowerShell

Health Check Script

Database Health Monitoring

PostgreSQL (Catalog & Basket)

Redis (Basket Cache)

SQL Server (Ordering)

Container Health Monitoring

Docker Container Status

Resource Usage Monitoring

Application Logging

Log Levels

Viewing Logs

Log Patterns to Monitor

Success Patterns

Error Patterns

Structured Logging

RabbitMQ Monitoring

Management UI

Key Metrics to Monitor

Queues

Exchanges

Connections

CLI Monitoring

Testing Message Flow

Service-Specific Monitoring

Catalog API

Basket API

Ordering API

Discount gRPC Service

API Gateway (YARP)

Performance Monitoring

Response Time Monitoring

Database Query Performance

Alerting Considerations

Critical Alerts

Warning Alerts

Production Monitoring Recommendations

Next Steps