Logging

Overview

Masar Eagle uses Grafana Loki for centralized log aggregation, with structured logging through OpenTelemetry. All logs are automatically enriched with context and sent to Loki via the OpenTelemetry Collector.

Loki Configuration

Server Setup

Loki is configured in AppHost.cs (src/aspire/AppHost/AppHost.cs:58-61):

IResourceBuilder<ContainerResource> loki = 
    builder.AddContainer("loki", "grafana/loki:latest")
    .WithBindMount("../loki/config.yaml", "/etc/loki/config.yaml", isReadOnly: true)
    .WithHttpEndpoint(targetPort: 3100, name: "http")
    .WithArgs("-config.file=/etc/loki/config.yaml");

Loki Settings

The configuration (src/aspire/loki/config.yaml) enables modern Loki features:

auth_enabled: false

server:
  http_listen_port: 3100

common:
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory
  replication_factor: 1
  path_prefix: /tmp/loki

schema_config:
  configs:
  - from: 2020-05-15
    store: tsdb
    object_store: filesystem
    schema: v13
    index:
      prefix: index_
      period: 24h

limits_config:
  volume_enabled: true
  discover_log_levels: true
  allow_structured_metadata: true

pattern_ingester:
  enabled: true

Key Features Explained

TSDB Schema: Modern time-series database for better query performance
Volume Enabled: Aggregate logs by volume for cost optimization
Log Level Discovery: Automatically detect log levels (INFO, WARN, ERROR)
Structured Metadata: Support for key-value metadata on log lines
Pattern Ingester: Automatic log pattern detection and extraction

Log Levels

Default Configuration

Log levels are configured per service in appsettings.json:

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning",
      "Microsoft.EntityFrameworkCore": "Warning",
      "System.Net.Http.HttpClient": "Warning",
      "Microsoft.Hosting.Lifetime": "Information"
    }
  }
}

Log Level Hierarchy

Trace

Extremely detailed diagnostic information. Only use in development for deep debugging.

Debug

Detailed flow information for debugging. Not enabled in production.

Information (Default)

General application flow. Key business events, state changes.

Warning

Unexpected events that don’t prevent operation. Client errors (4xx).

Error

Failures in current operation. Server errors (5xx), exceptions.

Critical

Application crashes, data loss, critical failures requiring immediate attention.

Structured Logging

OpenTelemetry Integration

Logging is configured in ServiceDefaults (src/aspire/ServiceDefaults/Extensions.cs:36-40):

builder.Logging.AddOpenTelemetry(logging =>
{
    logging.IncludeFormattedMessage = true;
    logging.IncludeScopes = true;
});

This ensures:

Logs include formatted message text
Log scopes are preserved for context
Automatic correlation with traces

Error Logging

The GlobalExceptionMiddleware (src/BuildingBlocks/Common/Middleware/GlobalExceptionMiddleware.cs:93-172) creates rich structured logs:

using (logger.BeginScope(new Dictionary<string, object?>
{
    ["ErrorId"] = errorId,
    ["ExceptionType"] = exception.GetType().FullName,
    ["ExceptionName"] = exception.GetType().Name,
    ["RequestPath"] = requestPath,
    ["RequestMethod"] = requestMethod,
    ["StatusCode"] = statusCode,
    ["ErrorCategory"] = errorCategory,
    ["ServiceName"] = serviceName,
    ["UserId"] = userId ?? "Anonymous",
    ["UserRole"] = userRole ?? "None",
    ["TraceId"] = traceId,
    ["QueryString"] = queryString,
    ["UserAgent"] = userAgent,
    ["Referer"] = referer,
    ["RemoteIp"] = remoteIp,
    ["ErrorDurationMs"] = elapsed.TotalMilliseconds,
    ["HasInnerException"] = exception.InnerException != null
}))
{
    if (isClientError)
    {
        logger.LogWarning(
            exception,
            "[ErrorId: {ErrorId}] Client error: {ExceptionType} - {ExceptionMessage}");
    }
    else
    {
        logger.LogError(
            exception,
            "[ErrorId: {ErrorId}] Server error: {ExceptionType} - {ExceptionMessage}");
    }
}

Error Categories

Errors are automatically categorized (src/BuildingBlocks/Common/Middleware/GlobalExceptionMiddleware.cs:174-189):

Exception Type	Category	Log Level
`ArgumentException`, `ArgumentNullException`	ValidationError	Warning
`InvalidOperationException`	BusinessLogicError	Warning
`UnauthorizedAccessException`	AuthenticationError	Warning
`UserNotFoundException`	AuthenticationError	Warning
`UserForbiddenException`	AuthorizationError	Warning
`KeyNotFoundException`	NotFoundError	Warning
`TimeoutException`	TimeoutError	Error
`HttpRequestException`	ExternalServiceError	Error
Other exceptions	UnexpectedError	Error

Request/Response Logging

Gateway Logging

The Gateway service includes detailed request/response logging (src/services/Gateway.Api/appsettings.json:9-29):

"RequestResponse": {
  "LogRequests": true,
  "LogResponses": true,
  "LogRequestBody": false,
  "LogResponseBody": false,
  "MaxBodySize": 1024,
  "ExcludedPaths": [
    "/health",
    "/metrics",
    "/ready",
    "/live"
  ],
  "ExcludedHeaders": [
    "Authorization",
    "Cookie",
    "X-Api-Key"
  ],
  "RequestLogLevel": "Information",
  "ResponseLogLevel": "Information",
  "ErrorLogLevel": "Error"
}

Request Tracking

The RequestResponseLoggingMiddleware (src/services/Gateway.Api/Middleware/RequestResponseLoggingMiddleware.cs:15-60) adds:

string requestId = GenerateRequestId(context);
context.Items["RequestId"] = requestId;
context.Response.Headers["X-Request-Id"] = requestId;

var stopwatch = Stopwatch.StartNew();

await requestLogger.LogRequestAsync(context, requestId);
await _next(context);

stopwatch.Stop();
await requestLogger.LogResponseAsync(
    context, requestId, statusCode, 
    responseTimeMs, destinationService, 
    destinationUrl, exception);

Every request gets a unique X-Request-Id header for end-to-end tracing across services.

Log Queries

LogQL Examples

Filter by Service

{service_name="user"}

Error Logs Only

{service_name=~"user|trip|gateway"} |= "ERROR"

Specific Error ID

{service_name=~".*"} | json | ErrorId="A1B2C3D4E5F6G7H8"

User Activity

{service_name=~".*"} | json | UserId="12345"

High Latency Requests

{service_name="gateway"} | json | ErrorDurationMs > 1000

Rate of 5xx Errors

rate({service_name="gateway"} | json | StatusCode >= 500 [5m])

Top Error Types

topk(10, sum by (ExceptionType) (
  count_over_time({service_name=~".*"} | json | level="error" [1h])
))

Pattern Matching

Loki’s pattern ingester automatically detects common log patterns:

{service_name="user"} | pattern

Example detected patterns:

<_> Client error: <exception_type> - <message> | Path: <method> <path>
<_> Server error: <exception_type> - <message> | Path: <method> <path>

Log Retention

The current configuration uses local filesystem storage (/tmp/loki). For production:

Configure external object storage (S3, GCS, Azure Blob)
Set retention policies
Enable compaction

Best Practices

Use Structured Fields

Always use structured logging with key-value pairs instead of string interpolation:

// Good
logger.LogInformation("User {UserId} created trip {TripId}", userId, tripId);

// Bad
logger.LogInformation($"User {userId} created trip {tripId}");

Include Error IDs

Always include error IDs in exception responses for correlation:

problemDetails.Extensions["errorId"] = errorId;

Don't Log Secrets

Ensure sensitive headers are excluded:

"ExcludedHeaders": ["Authorization", "Cookie", "X-Api-Key"]

Use Log Scopes

Group related log entries with scopes:

using (logger.BeginScope(new { TripId = tripId }))
{
    // All logs here include TripId
}

Troubleshooting Logs

No Logs Appearing

Check OTLP Endpoint

Verify services have the correct OpenTelemetry endpoint:

echo $OTEL_EXPORTER_OTLP_ENDPOINT
# Should be: http://otelcollector:4317

Verify Collector

Check OpenTelemetry Collector logs:

docker logs otelcollector

Check Loki

Verify Loki is receiving logs:

curl http://localhost:3100/ready
curl http://localhost:3100/metrics | grep loki_ingester_streams

Missing Structured Fields

Ensure IncludeScopes is enabled in OpenTelemetry logging configuration.

Log Level Too Verbose

Adjust in appsettings.json or via environment variable:

LOGGING__LOGLEVEL__DEFAULT=Warning

Getting Started

Setup & Deployment

Core Concepts

Services

Features

Development

Operations

Overview

Loki Configuration

Server Setup

Loki Settings

Log Levels

Default Configuration

Log Level Hierarchy

Structured Logging

OpenTelemetry Integration

Error Logging

Error Categories

Request/Response Logging

Gateway Logging

Request Tracking

Log Queries

LogQL Examples

Pattern Matching

Log Retention

Best Practices

Use Structured Fields

Include Error IDs

Don't Log Secrets

Use Log Scopes

Troubleshooting Logs

No Logs Appearing

Missing Structured Fields

Log Level Too Verbose

Next Steps

Monitoring

Troubleshooting

Build docs developers (and LLMs) love

Getting Started

Setup & Deployment

Core Concepts

Services

Features

Development

Operations

​Overview

​Loki Configuration

​Server Setup

​Loki Settings

​Log Levels

​Default Configuration

​Log Level Hierarchy

​Structured Logging

​OpenTelemetry Integration

​Error Logging

​Error Categories

​Request/Response Logging

​Gateway Logging

​Request Tracking

​Log Queries

​LogQL Examples

​Pattern Matching

​Log Retention

​Best Practices

Use Structured Fields

Include Error IDs

Don't Log Secrets

Use Log Scopes

​Troubleshooting Logs

​No Logs Appearing

​Missing Structured Fields

​Log Level Too Verbose

​Next Steps

Monitoring

Troubleshooting

Build docs developers (and LLMs) love

Overview

Loki Configuration

Server Setup

Loki Settings

Log Levels

Default Configuration

Log Level Hierarchy

Structured Logging

OpenTelemetry Integration

Error Logging

Error Categories

Request/Response Logging

Gateway Logging

Request Tracking

Log Queries

LogQL Examples

Pattern Matching

Log Retention

Best Practices

Troubleshooting Logs

No Logs Appearing

Missing Structured Fields

Log Level Too Verbose

Next Steps