Monitoring and Logging

CVAT provides built-in monitoring and logging capabilities through an integrated analytics stack based on ClickHouse, Vector, and Grafana.

Analytics Stack Overview

The analytics infrastructure consists of:

ClickHouse: Time-series database for storing events and logs
Vector: Log aggregation and forwarding
Grafana: Visualization and dashboards
Traefik: Access logs and routing

Defined in docker-compose.yml:332-419.

Enabling Analytics

Enable Analytics in Docker Compose

Set the environment variable in docker-compose.yml:99 or in your override file:

services:
  cvat_server:
    environment:
      CVAT_ANALYTICS: 1

Alternatively, set as a shell variable:

export CVAT_ANALYTICS=1
docker compose up -d

Django Settings

Analytics are controlled by the ANALYTICS_ENABLED setting in cvat/settings/base.py:177-180:

ANALYTICS_ENABLED = to_bool(os.getenv("CVAT_ANALYTICS", False))

if ANALYTICS_ENABLED:
    INSTALLED_APPS += ["cvat.apps.log_viewer"]

ClickHouse Configuration

ClickHouse stores analytics events and logs.

Environment Variables

Defined in docker-compose.yml:31-36:

x-clickhouse-env: &clickhouse-env
  CLICKHOUSE_HOST: clickhouse
  CLICKHOUSE_PORT: 8123
  CLICKHOUSE_DB: cvat
  CLICKHOUSE_USER: user
  CLICKHOUSE_PASSWORD: user

Django Configuration

ClickHouse settings in cvat/settings/base.py:701-709:

CLICKHOUSE = {
    "events": {
        "NAME": os.getenv("CLICKHOUSE_DB", "cvat"),
        "HOST": os.getenv("CLICKHOUSE_HOST", "localhost"),
        "PORT": os.getenv("CLICKHOUSE_PORT", 8123),
        "USER": os.getenv("CLICKHOUSE_USER", "user"),
        "PASSWORD": os.getenv("CLICKHOUSE_PASSWORD", "user"),
    }
}

ClickHouse Container

Defined in docker-compose.yml:332-343:

cvat_clickhouse:
  container_name: cvat_clickhouse
  image: clickhouse/clickhouse-server:23.11-alpine
  restart: always
  environment:
    <<: *clickhouse-env
  volumes:
    - cvat_events_db:/var/lib/clickhouse/
  networks:
    cvat:
      aliases:
        - clickhouse

Vector Log Aggregation

Vector collects logs from Django and forwards them to ClickHouse.

Vector Configuration

Source: components/analytics/vector/vector.toml:1-47

data_dir = "/vector-data-dir"

[sources.http-events]
type = "http_server"
address = "0.0.0.0:8282"
encoding = "json"

[sinks.clickhouse]
inputs = [ "http-events" ]
type = "clickhouse"
database = "${CLICKHOUSE_DB}"
table = "events"
auth.strategy = "basic"
auth.user = "${CLICKHOUSE_USER}"
auth.password = "${CLICKHOUSE_PASSWORD}"
endpoint = "http://${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}"

Event Fields

Vector forwards these fields to ClickHouse:

scope: Event scope/category
obj_name: Object name
obj_id: Object ID
obj_val: Object value
source: Event source
timestamp: Event timestamp
count: Event count
duration: Event duration
project_id: Associated project ID
task_id: Associated task ID
job_id: Associated job ID
user_id: User ID
user_name: Username
user_email: User email
org_id: Organization ID
org_slug: Organization slug
payload: Additional event data
access_token_id: Access token ID

Vector Container

Defined in docker-compose.yml:345-364:

cvat_vector:
  container_name: cvat_vector
  image: timberio/vector:0.26.0-alpine
  restart: always
  depends_on:
    - cvat_clickhouse
  environment:
    <<: *clickhouse-env
  volumes:
    - ./components/analytics/vector/vector.toml:/etc/vector/vector.toml:ro
    - type: tmpfs
      target: /vector-data-dir
  networks:
    cvat:
      aliases:
        - vector

Django Logging Configuration

Logging setup in cvat/settings/base.py:481-553:

Log Handler

vector_log_handler = os.getenv("VECTOR_EVENT_HANDLER", "AsynchronousLogstashHandler")

LOGGING = {
    "handlers": {
        "vector": {
            "level": "INFO",
            "class": f"logstash_async.handler.{vector_log_handler}",
            "formatter": "vector",
            "transport": "logstash_async.transport.HttpTransport",
            "ssl_enable": False,
            "ssl_verify": False,
            "host": os.getenv("DJANGO_LOG_SERVER_HOST", "localhost"),
            "port": os.getenv("DJANGO_LOG_SERVER_PORT", 8282),
            "version": 1,
            "message_type": "django",
            "database_path": EVENTS_LOCAL_DB_FILE,
        },
    },
}

Log Files

File-based logging handlers:

"server_file": {
    "class": "logging.handlers.RotatingFileHandler",
    "level": "DEBUG",
    "filename": LOGS_ROOT / "cvat_server.log",
    "formatter": "standard",
    "maxBytes": 1024 * 1024 * 50,  # 50 MB
    "backupCount": 5,
},
"dataset_handler": {
    "class": "logging.handlers.RotatingFileHandler",
    "level": "DEBUG",
    "filename": LOGS_ROOT / "cvat_server_dataset.log",
    "formatter": "standard",
    "maxBytes": 1024 * 1024 * 50,  # 50 MB
    "backupCount": 3,
},

Logs are stored in the cvat_logs volume mounted at /home/django/logs.

Log Level Configuration

DJANGO_LOG_LEVEL=DEBUG

See cvat/settings/base.py:532.

Grafana Dashboards

Grafana provides visualization of analytics data stored in ClickHouse.

Accessing Grafana

Grafana is accessible at:

http://your-cvat-host:8080/analytics

Traefik routing defined in components/analytics/grafana_conf.yml.

Grafana Configuration

Grafana container in docker-compose.yml:366-419:

cvat_grafana:
  image: grafana/grafana-oss:10.1.2
  restart: always
  environment:
    <<: *clickhouse-env
    GF_PATHS_PROVISIONING: /etc/grafana/provisioning
    GF_AUTH_BASIC_ENABLED: false
    GF_AUTH_ANONYMOUS_ENABLED: true
    GF_AUTH_ANONYMOUS_ORG_ROLE: Admin
    GF_AUTH_DISABLE_LOGIN_FORM: true
    GF_SERVER_ROOT_URL: http://${CVAT_HOST:-localhost}/analytics
    GF_INSTALL_PLUGINS: 'grafana-clickhouse-datasource 4.8.2'
    GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH: /var/lib/grafana/dashboards/all_events.json
  volumes:
    - ./components/analytics/grafana/dashboards/:/var/lib/grafana/dashboards/:ro

Anonymous Access

By default, Grafana allows anonymous access with admin role:

GF_AUTH_BASIC_ENABLED: false
GF_AUTH_ANONYMOUS_ENABLED: true
GF_AUTH_ANONYMOUS_ORG_ROLE: Admin
GF_AUTH_DISABLE_LOGIN_FORM: true

To restrict access, modify these settings in your docker-compose.override.yml:

services:
  cvat_grafana:
    environment:
      GF_AUTH_ANONYMOUS_ENABLED: false
      GF_AUTH_BASIC_ENABLED: true
      GF_AUTH_DISABLE_LOGIN_FORM: false
      GF_SECURITY_ADMIN_PASSWORD: your-secure-password

Pre-built Dashboards

CVAT includes three dashboards in components/analytics/grafana/dashboards/:

all_events.json: All events overview (default home dashboard)
management.json: Management metrics
monitoring.json: System monitoring

Custom Dashboards

To add custom dashboards:

Create a dashboard JSON file
Place it in components/analytics/grafana/dashboards/
Restart Grafana:

docker compose restart cvat_grafana

The dashboard will be automatically provisioned.

Traefik Access Logs

Traefik logs all HTTP requests in JSON format.

Log Configuration

Defined in docker-compose.yml:275-287:

traefik:
  environment:
    TRAEFIK_ACCESSLOG_FORMAT: json
    TRAEFIK_ACCESSLOG_FIELDS_DEFAULTMODE: drop
    TRAEFIK_ACCESSLOG_FIELDS_NAMES_ClientHost: keep
    TRAEFIK_ACCESSLOG_FIELDS_NAMES_DownstreamContentSize: keep
    TRAEFIK_ACCESSLOG_FIELDS_NAMES_DownstreamStatus: keep
    TRAEFIK_ACCESSLOG_FIELDS_NAMES_Duration: keep
    TRAEFIK_ACCESSLOG_FIELDS_NAMES_RequestHost: keep
    TRAEFIK_ACCESSLOG_FIELDS_NAMES_RequestMethod: keep
    TRAEFIK_ACCESSLOG_FIELDS_NAMES_RequestPath: keep
    TRAEFIK_ACCESSLOG_FIELDS_NAMES_RequestPort: keep
    TRAEFIK_ACCESSLOG_FIELDS_NAMES_RequestProtocol: keep
    TRAEFIK_ACCESSLOG_FIELDS_NAMES_RouterName: keep
    TRAEFIK_ACCESSLOG_FIELDS_NAMES_StartUTC: keep

Log Rotation

Traefik logs are rotated automatically:

logging:
  driver: "json-file"
  options:
    max-size: 100m
    max-file: "10"

Logs are limited to 100MB per file with a maximum of 10 files.

Viewing Traefik Logs

docker compose logs -f traefik

Container Health Monitoring

CVAT includes health check endpoints.

Health Check Apps

Installed in cvat/settings/base.py:103-106:

"health_check",
"health_check.cache",
"health_check.db",
"health_check.contrib.psutil",

Health Check Configuration

Disk usage threshold in cvat/settings/base.py:788:

HEALTH_CHECK = {
    "DISK_USAGE_MAX": int(os.getenv("CVAT_HEALTH_DISK_USAGE_MAX", 90))
}

Health Check Endpoint

Access health status:

curl http://localhost:8080/api/health/

Response includes:

Database connectivity
Cache availability
Disk usage
System resources (CPU, memory)

Monitoring with External Tools

Prometheus Integration

To export metrics to Prometheus:

Install django-prometheus:

RUN pip install django-prometheus

Add to Django settings:

INSTALLED_APPS += ['django_prometheus']

MIDDLEWARE = [
    'django_prometheus.middleware.PrometheusBeforeMiddleware',
] + MIDDLEWARE + [
    'django_prometheus.middleware.PrometheusAfterMiddleware',
]

Expose metrics endpoint in docker-compose.override.yml:

services:
  cvat_server:
    ports:
      - "9090:9090"

Datadog Integration

Add Datadog agent to your compose file:

services:
  datadog:
    image: datadog/agent:latest
    environment:
      - DD_API_KEY=${DATADOG_API_KEY}
      - DD_SITE=datadoghq.com
      - DD_LOGS_ENABLED=true
      - DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /proc/:/host/proc/:ro
      - /sys/fs/cgroup/:/host/sys/fs/cgroup:ro

Sentry Error Tracking

Add Sentry SDK to track errors:

Install sentry-sdk:

RUN pip install sentry-sdk

Configure in custom settings:

import sentry_sdk
from sentry_sdk.integrations.django import DjangoIntegration

sentry_sdk.init(
    dsn="https://[email protected]/project-id",
    integrations=[DjangoIntegration()],
    traces_sample_rate=1.0,
    send_default_pii=True
)

Log Management

Accessing Container Logs

# All services
docker compose logs

# Specific service
docker compose logs cvat_server

# Follow logs
docker compose logs -f cvat_server

# Last N lines
docker compose logs --tail=100 cvat_server

Log Files Location

Application logs are stored in the cvat_logs volume:

docker exec cvat_server ls -lh /home/django/logs/

Files:

cvat_server.log: Main application log
cvat_server_dataset.log: Dataset operations log

Extracting Logs from Volume

docker run --rm \
  --volumes-from cvat_server \
  -v $(pwd):/backup \
  ubuntu tar -czvf /backup/cvat_logs.tar.gz /home/django/logs

Log Retention

Rotating file handlers automatically manage log retention:

Server logs: 5 files × 50 MB = 250 MB max
Dataset logs: 3 files × 50 MB = 150 MB max

Performance Monitoring

Database Performance

Monitor PostgreSQL performance:

# Connection stats
docker exec cvat_db psql -U root -d cvat -c "
  SELECT count(*), state FROM pg_stat_activity GROUP BY state;"

# Slow queries
docker exec cvat_db psql -U root -d cvat -c "
  SELECT query, calls, total_time, mean_time 
  FROM pg_stat_statements 
  ORDER BY total_time DESC LIMIT 10;"

Redis Performance

Monitor Redis:

# In-memory Redis stats
docker exec cvat_redis_inmem redis-cli INFO stats

# Kvrocks stats
docker exec cvat_redis_ondisk redis-cli INFO stats

Container Resource Usage

# Real-time stats
docker stats

# Specific container
docker stats cvat_server

Troubleshooting

Analytics Not Working

Check if analytics are enabled:

docker exec cvat_server python -c "from django.conf import settings; print(settings.ANALYTICS_ENABLED)"

Verify ClickHouse is running:

docker compose ps cvat_clickhouse

Check Vector is forwarding logs:

docker compose logs cvat_vector

Grafana Connection Issues

Test ClickHouse connectivity from Grafana:

docker exec cvat_grafana wget -O- http://clickhouse:8123

Verify ClickHouse credentials:

docker exec cvat_clickhouse clickhouse-client --user=user --password=user --query="SELECT 1"

High Disk Usage

Check volume sizes:

docker system df -v

Clear ClickHouse data:

docker exec cvat_clickhouse clickhouse-client --query="TRUNCATE TABLE cvat.events"

Best Practices

Enable analytics: Always enable analytics in production for troubleshooting
Monitor disk space: Set up alerts for disk usage thresholds
Backup analytics data: Include ClickHouse in backup procedures
Secure Grafana: Disable anonymous access in production
Rotate logs: Ensure log rotation is working to prevent disk exhaustion
Monitor container health: Set up external monitoring of health endpoints
Review dashboards regularly: Check Grafana dashboards for anomalies

Additional Resources

ClickHouse Documentation
Vector Documentation
Grafana Documentation
Traefik Documentation
Source: components/analytics/vector/vector.toml
Source: components/analytics/grafana/dashboards/
Source: docker-compose.yml:332-419

Installation

Administration

Serverless Functions

​Analytics Stack Overview

​Enabling Analytics

​Enable Analytics in Docker Compose

​Django Settings

​ClickHouse Configuration

​Environment Variables

​Django Configuration

​ClickHouse Container

​Vector Log Aggregation

​Vector Configuration

​Event Fields

​Vector Container

​Django Logging Configuration

​Log Handler

​Log Files

​Log Level Configuration

​Grafana Dashboards

​Accessing Grafana

​Grafana Configuration

​Anonymous Access

​Pre-built Dashboards

​Custom Dashboards

​Traefik Access Logs

​Log Configuration

​Log Rotation

​Viewing Traefik Logs

​Container Health Monitoring

​Health Check Apps

​Health Check Configuration

​Health Check Endpoint

​Monitoring with External Tools

​Prometheus Integration

​Datadog Integration

​Sentry Error Tracking

​Log Management

​Accessing Container Logs

​Log Files Location

​Extracting Logs from Volume

​Log Retention

​Performance Monitoring

​Database Performance

​Redis Performance

​Container Resource Usage

​Troubleshooting

​Analytics Not Working

​Grafana Connection Issues

​High Disk Usage

​Best Practices

​Additional Resources

Build docs developers (and LLMs) love