Log Viewer

​

Overview

​

Security Warning

​

Required Security Measures

• ✅ Deploy behind a reverse proxy with authentication (e.g., nginx with basic auth)
• ✅ Use firewall rules to restrict access to trusted IP ranges only
• ✅ Never expose log viewer ports directly to the internet
• ✅ Monitor access to log endpoints in your infrastructure logs
• ✅ Consider VPN-only access for maximum security

​

Data Exposure Risk

• 🔑 GitHub Personal Access Tokens and API credentials
• 👤 User information and GitHub usernames
• 📋 Repository details and webhook payloads
• 🔒 Internal system information and error details

​

Network-Level Security

1. VPN Access: Deploy behind corporate VPN for internal-only access
2. Reverse Proxy Authentication: Use nginx/Apache with HTTP Basic Auth:

location /logs {
    auth_basic "Webhook Logs";
    auth_basic_user_file /etc/nginx/.htpasswd;
    proxy_pass http://webhook-server:5000;
}

3. Firewall Rules: Restrict access to webhook server port to specific IP ranges
4. Network Segmentation: Deploy in isolated network segments

​

Performance & Scalability

​

Memory-Optimized Streaming

• Constant Memory Usage: Handles log files of any size with consistent memory footprint
• Early Filtering: Reduces data transfer by filtering at the source before transmission
• Streaming Processing: Real-time log processing without loading entire files into memory
• 90% Memory Reduction: Optimized for enterprise environments with gigabytes of log data
• Sub-second Response Times: Fast query responses even with large datasets

​

Performance Benchmarks

• 90% reduction in memory usage compared to bulk loading
• Sub-second response times for filtered queries on multi-GB log files
• Constant memory footprint regardless of log file size
• Real-time streaming with less than 100ms latency for new log entries

​

Technical Architecture

Log File → Streaming Parser → Early Filter → Chunked Processing → Client
    ↓            ↓               ↓              ↓              ↓
Real-time    Line-by-line    Apply filters   Small batches   Progressive UI
processing   microsecond     before load     (100-1000       updates
             timestamps                       entries)

• Streaming Parser: Reads log files line-by-line instead of loading entire files
• Early Filtering: Applies search criteria during parsing to reduce memory usage
• Chunked Responses: Delivers results in small batches for responsive UI
• Automatic Cleanup: Releases processed data immediately after transmission

​

Accessing the Log Viewer

​

Enable Log Viewer

# Environment variable
export ENABLE_LOG_SERVER=true

services:
  github-webhook-server:
    environment:
      - ENABLE_LOG_SERVER=1

​

Web Interface

http://your-server:5000/logs

​

Core Features

• 🔍 Real-time log streaming via WebSocket connections with intelligent buffering
• 📊 Advanced filtering by hook ID, PR number, repository, user, log level, and text search
• 🎨 Dark/light theme support with automatic preference saving
• 📈 PR flow visualization showing webhook processing stages and timing
• 📥 JSON export functionality for log analysis and external processing
• 🎯 Color-coded log levels for quick visual identification
• ⚡ Progressive loading with pagination for large datasets
• 🔄 Auto-refresh with configurable intervals
• 🎛️ Advanced query builder for complex log searches

​

Web Interface Features

​

Filtering Controls

• Hook ID: GitHub delivery ID for tracking specific webhook calls
• PR Number: Filter by pull request number
• Repository: Filter by repository name (org/repo format)
• User: Filter by GitHub username
• Log Level: Filter by severity level (DEBUG, INFO, WARNING, ERROR)
• Search: Free text search across log messages
• Time Range: Filter by start and end time (ISO 8601 format)

PR Number: 123
Repository: my-org/my-repo
Log Level: ERROR
Search: "tox failed"

​

Real-time Features

• Live Updates: WebSocket connection for real-time log streaming
• Auto-refresh: Historical logs refresh when filters change
• Connection Status: Visual indicator for WebSocket connection status
• Progressive Loading: Smooth scrolling with automatic pagination

​

Theme Support

• Dark/Light Modes: Toggle between themes with automatic preference saving
• Responsive Design: Works on desktop and mobile devices
• Keyboard Shortcuts: Quick access to common functions

​

Log Level Color Coding

• 🟢 INFO (Green): Successful operations and informational messages
• 🟡 WARNING (Yellow): Warning messages that need attention
• 🔴 ERROR (Red): Error messages requiring immediate action
• ⚪ DEBUG (Gray): Technical debug information

​

API Endpoints

​

Get Historical Log Entries

GET /logs/api/entries

• hook_id (string): Filter by GitHub delivery ID
• pr_number (integer): Filter by pull request number
• repository (string): Filter by repository name (e.g., “org/repo”)
• event_type (string): Filter by GitHub event type
• github_user (string): Filter by GitHub username
• level (string): Filter by log level (DEBUG, INFO, WARNING, ERROR)
• start_time (string): Start time filter (ISO 8601 format)
• end_time (string): End time filter (ISO 8601 format)
• search (string): Free text search in log messages
• limit (integer): Maximum entries to return (1-1000, default: 100)
• offset (integer): Pagination offset (default: 0)

curl "http://localhost:5000/logs/api/entries?pr_number=123&level=ERROR&limit=50"

{
  "entries": [
    {
      "timestamp": "2025-01-30T10:30:00.123000",
      "level": "INFO",
      "logger_name": "GithubWebhook",
      "message": "Processing webhook for repository: my-org/my-repo",
      "hook_id": "abc123-def456",
      "event_type": "pull_request",
      "repository": "my-org/my-repo",
      "pr_number": 123,
      "github_user": "username"
    }
  ],
  "entries_processed": 1500,
  "filtered_count_min": 25,
  "limit": 50,
  "offset": 0
}

​

Export Logs

GET /logs/api/export

• format (string): Export format - only “json” is supported
• limit (integer): Maximum entries to export (max 50,000, default: 10,000)

curl "http://localhost:5000/logs/api/export?format=json&pr_number=123" -o logs.json

​

WebSocket Real-time Streaming

ws://your-server:5000/logs/ws

const ws = new WebSocket("ws://localhost:5000/logs/ws?level=ERROR");
ws.onmessage = function(event) {
  const logEntry = JSON.parse(event.data);
  console.log("New error log:", logEntry);
};

​

PR Flow Visualization

GET /logs/api/pr-flow/{identifier}

• identifier: Hook ID (e.g., “abc123”) or PR number (e.g., “123”)

curl "http://localhost:5000/logs/api/pr-flow/123"

{
  "identifier": "123",
  "stages": [
    {
      "name": "Webhook Received",
      "timestamp": "2025-01-30T10:30:00.123000",
      "duration_ms": null
    },
    {
      "name": "Validation Complete",
      "timestamp": "2025-01-30T10:30:00.245000",
      "duration_ms": 122
    }
  ],
  "total_duration_ms": 2500,
  "success": true
}

​

Usage Examples

​

Monitor Specific PR

# View all logs for PR #123
curl "http://localhost:5000/logs/api/entries?pr_number=123"

​

Track Webhook Processing

# Follow specific webhook delivery
curl "http://localhost:5000/logs/api/entries?hook_id=abc123-def456"

​

Debug Error Issues

# Export all error logs for analysis
curl "http://localhost:5000/logs/api/export?format=json&level=ERROR" -o errors.json

​

Monitor Repository Activity

# Watch real-time activity for specific repository
# Connect WebSocket to: ws://localhost:5000/logs/ws?repository=my-org/my-repo

​

Search for Specific Errors

# Find all "connection timeout" errors
curl "http://localhost:5000/logs/api/entries?search=connection%20timeout&level=ERROR"

​

Troubleshooting

​

WebSocket Connection Issues

• Check firewall rules for WebSocket traffic
• Verify server is accessible on specified port
• Ensure WebSocket upgrades are allowed by reverse proxy
• Check browser console for connection errors

​

Missing Log Data

• Verify log file permissions and paths
• Check if log directory exists and is writable
• Ensure log parser patterns match your log format
• Verify ENABLE_LOG_SERVER is set to true

​

Performance Issues

• Large Result Sets: Reduce filter result sets using specific time ranges or repositories
• Memory Usage: The streaming architecture automatically handles large datasets efficiently
• Query Optimization: Use specific filters (hook_id, pr_number) for fastest responses
• File Size Management: Consider log file rotation for easier management
• Network Latency: Use pagination for mobile or slow connections

​

Log File Format Issues

• Verify log file format matches expected JSON structure
• Check for corrupted log entries
• Ensure UTF-8 encoding
• Review log rotation configuration

​

Log File Format

{
  "hook_id": "github-delivery-id",
  "event_type": "pull_request",
  "pr": {"number": 968, "title": "Add new feature"},
  "timing": {
    "started_at": "2026-01-05T10:30:00.123Z",
    "duration_ms": 7712
  },
  "workflow_steps": {
    "clone_repository": {"status": "completed", "duration_ms": 4823}
  },
  "token_spend": 4,
  "success": true
}

​

Best Practices

​

Security

1. Never expose to internet: Deploy behind VPN or with authentication
2. Monitor access: Track who accesses log viewer endpoints
3. Rotate logs: Implement log retention policies
4. Encrypt transit: Use TLS/SSL for all connections
5. Audit regularly: Review access logs and security configurations

​

Performance

1. Use specific filters: Narrow down results with precise filters
2. Limit result sets: Use pagination for large datasets
3. Enable log rotation: Keep individual log files manageable
4. Monitor resource usage: Track memory and CPU usage
5. Optimize queries: Use hook_id or pr_number for fastest lookups

​

Operational

1. Regular backups: Backup log files for compliance and auditing
2. Monitor disk space: Ensure adequate storage for logs
3. Set retention policies: Define how long to keep logs
4. Document procedures: Create runbooks for common tasks
5. Test recovery: Verify log recovery procedures work