Grafana Dashboards

​

Overview

​

Dashboard Architecture

• Persistence: Enabled with 10Gi StatefulSet
• Plugins: grafana-piechart-panel
• Ingress: Available at grafana.pennlabs.org with TLS
• Data Source: Prometheus server in monitoring namespace

dashboardProviders:
  dashboardproviders.yaml:
    apiVersion: 1
    providers:
      - name: "default"
        orgId: 1
        type: file
        editable: true
        options:
          path: /var/lib/grafana/dashboards/default

​

Available Dashboards

​

1. Traefik Dashboard

• Request rate and response times
• HTTP status code breakdown (2xx, 4xx, 5xx)
• Backend health and availability
• Request duration percentiles
• Active connections

• Diagnose slow API responses
• Identify traffic spikes
• Monitor ingress health
• Track error rates by endpoint

​

2. Pod Dashboard

• Pod status (Running, Pending, Failed)
• Container restarts
• Resource usage (CPU, Memory)
• Pod age and uptime
• Node distribution

• Quick overview of cluster health
• Identify pods with issues
• Monitor deployment rollouts
• Check resource utilization
• Detect crash loops

​

3. Pod Alerting Dashboard

• High CPU usage
• High memory usage
• Excessive container restarts
• Pod crash loops
• Pods stuck in Pending state

​

4. Cert Manager Dashboard

• Certificate expiration dates
• Certificate renewal status
• ACME challenge success/failure
• Certificate issuance time
• Ready vs Not Ready certificates

• Prevent certificate expiration
• Monitor cert-manager health
• Troubleshoot certificate issues
• Track certificate renewal process

​

5. Node Exporter Dashboard

• CPU usage and load average
• Memory and swap usage
• Disk I/O and space
• Network traffic
• System uptime

• Monitor node health
• Identify resource bottlenecks
• Plan capacity upgrades
• Diagnose performance issues

​

Dashboard Configuration

resource "helm_release" "grafana" {
  name       = "grafana"
  repository = "https://charts.helm.sh/stable"
  chart      = "grafana"
  version    = "5.1.4"
  
  values = [file("helm/grafana.yaml")]
}

dashboards:
  default:
    node-exporter:
      gnetId: 1860
      revision: 19
    cert-manager:
      url: https://raw.githubusercontent.com/pennlabs/infrastructure/master/grafana-dashboards/cert-manager.json
    traefik:
      url: https://raw.githubusercontent.com/pennlabs/infrastructure/master/grafana-dashboards/traefik.json
    pod-alerting-dashboard:
      url: https://raw.githubusercontent.com/pennlabs/infrastructure/master/grafana-dashboards/pod-alerting-dashboard.json
    pod-dashboard:
      url: https://raw.githubusercontent.com/pennlabs/infrastructure/master/grafana-dashboards/pod-dashboard.json

1. Edit the JSON file in grafana-dashboards/
2. Commit and push to the master branch
3. Grafana will reload the dashboard automatically

​

Data Source Configuration

datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
      - name: Prometheus
        type: prometheus
        url: http://prometheus-server.monitoring
        access: proxy
        orgId: 1

• Namespace: monitoring
• Service: prometheus-server
• Port: 80 (HTTP)

​

Alerting Configuration

notifiers:
  notifiers.yaml:
    notifiers:
      - name: Slack
        type: slack
        uid: slack
        org_id: 1
        is_default: true
        send_reminder: false
        settings:
          url: ${SLACK_NOTIFICATION_URL}

module "vault" {
  source = "./modules/vault"
  GF_SLACK_URL = var.GF_SLACK_URL
  # ... other config
}

envFromSecret: "grafana"

• ADMIN_USER - Grafana admin username
• ADMIN_PASSWORD - Grafana admin password
• SLACK_NOTIFICATION_URL - Slack webhook for alerts

​

Using the Dashboards

​

Accessing Grafana

1. Navigate to https://grafana.pennlabs.org
2. Click “Sign in with GitHub”
3. Authorize the Grafana app
4. You’ll be logged in if you’re a member of pennlabs organization

grafana.ini:
  auth.github:
    enabled: true
    scopes: user:email,read:org
    allowed_organizations: pennlabs

​

Navigating Dashboards

• Click the Grafana logo (top left) to see all dashboards
• Use search to find specific dashboards
• Star frequently used dashboards for quick access

• Use the time picker (top right) to adjust time window
• Common ranges: Last 5m, 15m, 1h, 6h, 24h
• Use “Refresh” dropdown to auto-refresh

• Select specific namespace, pod, or service
• Variables filter the displayed metrics

​

Common Workflows

​

Checking Deployment Health

1. Open Pod Dashboard
2. Look for pods with status not “Running”
3. Check container restart counts
4. Review resource usage for the application

​

Investigating Slow Responses

1. Open Traefik Dashboard
2. Check request duration percentiles
3. Look for increased latency in specific backends
4. Correlate with pod resource usage

​

Monitoring Certificate Expiration

1. Open Cert Manager Dashboard
2. Check “Days Until Expiration” panel
3. Verify certificates are renewing automatically
4. Investigate any certificates in “Not Ready” state

​

Troubleshooting Node Issues

1. Open Node Exporter Dashboard
2. Select the problematic node
3. Check CPU, memory, and disk usage
4. Review network traffic and errors

​

Creating Custom Dashboards

​

In the UI (Temporary)

1. Click ”+” → “Dashboard”
2. Add panels with queries
3. Save the dashboard

​

In Code (Recommended)

1. Export the dashboard:
   • Click dashboard settings (gear icon)
   • Click “JSON Model”
   • Copy the JSON
2. Save to repository:
   cd grafana-dashboards/
   # Create new file
   vim my-dashboard.json
   # Paste and save
   
3. Add to Grafana config in terraform/helm/grafana.yaml:
   dashboards:
     default:
       my-dashboard:
         url: https://raw.githubusercontent.com/pennlabs/infrastructure/master/grafana-dashboards/my-dashboard.json
   
4. Apply the change:
   cd terraform/
   terraform apply -target=helm_release.grafana
   
5. Commit to Git:
   git add grafana-dashboards/my-dashboard.json terraform/helm/grafana.yaml
   git commit -m "Add custom dashboard for monitoring X"
   git push
   

​

Troubleshooting

​

Dashboard Not Loading

1. Check Prometheus is running:
   kubectl get pods -n monitoring -l app=prometheus
   
2. Verify data source connection:
   • Go to Configuration → Data Sources
   • Click “Prometheus”
   • Click “Test” button
3. Check time range is appropriate (not too far in past/future)

​

Alerts Not Firing

1. Check alert rules are defined in dashboard
2. Verify Slack webhook URL is correct:
   kubectl get secret grafana -o jsonpath='{.data.SLACK_NOTIFICATION_URL}' | base64 -d
   
3. Test notification channel in Grafana UI
4. Check Grafana logs for errors:
   kubectl logs -l app=grafana
   

​

Can’t Access Grafana

1. Verify you’re a member of pennlabs GitHub organization
2. Check GitHub OAuth app configuration
3. Ensure you’ve authorized the Grafana app
4. Try incognito/private browsing mode

​

Dashboard Changes Not Appearing

1. Verify file is committed to master branch
2. Check file URL is accessible (test in browser)
3. Restart Grafana pod:
   kubectl rollout restart deployment/grafana
   
4. Clear browser cache

​

Additional Resources

• Grafana Documentation
• Prometheus Documentation
• Grafana Dashboard Best Practices
• Alerting Guide
• Logging Documentation