Testing Guide

Testing is crucial for maintaining CVAT’s quality and reliability. This guide covers how to run existing tests and write new ones.

Test Structure

CVAT has several test suites:

Python REST API tests: tests/python/rest_api/
Python SDK tests: tests/python/sdk/
Python CLI tests: tests/python/cli/
Frontend unit tests: In component directories
E2E tests: tests/cypress/e2e/

Prerequisites

Before running tests:

Ensure Docker is running
Stop any existing CVAT instances that might use the same ports
Install Python dependencies:

pip install -r tests/python/requirements.txt

Install Node.js dependencies:

cd tests
yarn install

Running Python Tests

REST API Tests

The REST API tests use pytest and run against a real CVAT instance in Docker containers.

Run All Tests

pytest tests/python/

This command automatically:

Starts all necessary Docker containers
Runs the complete test suite
Cleans up after completion

Run Specific Test Files

pytest tests/python/rest_api/test_tasks.py

Run Specific Test Functions

pytest tests/python/rest_api/test_tasks.py::test_create_task

Run with Verbose Output

pytest tests/python/ -v

Run Tests in Parallel

pytest tests/python/ -n auto

SDK Tests

pytest tests/python/sdk/

CLI Tests

pytest tests/python/cli/

Keep Test Services Running

To keep the test environment running for debugging:

pytest tests/python/ --start-services

This starts the containers but doesn’t run tests, allowing you to:

Access the test instance at http://localhost:8080
Inspect the database
Test API endpoints manually

Stop the services:

pytest tests/python/ --stop-services

Running Frontend Tests

Unit Tests

CVAT uses Jest for frontend unit testing:

cd cvat-ui
yarn test

Run with coverage:

yarn test --coverage

Run specific test files:

yarn test TaskItem.test.tsx

Run in watch mode:

yarn test --watch

Running E2E Tests

E2E tests use Cypress to test the complete application flow.

Interactive Mode

Open Cypress Test Runner:

cd tests
yarn run cypress:open

This opens an interactive interface where you can:

Select specific tests to run
Watch tests execute in real-time
Debug failing tests
See screenshots and videos

Headless Mode

Run all E2E tests in headless mode:

cd tests
yarn run cypress:run

Run specific test file:

yarn run cypress:run --spec "cypress/e2e/actions_tasks.js"

E2E Test Configuration

Cypress configuration is in tests/cypress.config.js. Common settings:

{
  baseUrl: 'http://localhost:8080',
  video: true,
  screenshotOnRunFailure: true,
  defaultCommandTimeout: 10000
}

Writing Tests

Writing Python REST API Tests

REST API tests are located in tests/python/rest_api/.

Test Structure

import pytest
from http import HTTPStatus

class TestTasks:
    def test_create_task(self, admin_user):
        """Test creating a new task."""
        # Arrange
        task_spec = {
            "name": "Test Task",
            "labels": [{"name": "car"}]
        }
        
        # Act
        response = admin_user.post(
            "/api/tasks",
            json=task_spec
        )
        
        # Assert
        assert response.status_code == HTTPStatus.CREATED
        assert response.json()["name"] == "Test Task"
        assert len(response.json()["labels"]) == 1

    def test_get_task(self, admin_user, tasks):
        """Test retrieving a task by ID."""
        task_id = tasks[0]["id"]
        
        response = admin_user.get(f"/api/tasks/{task_id}")
        
        assert response.status_code == HTTPStatus.OK
        assert response.json()["id"] == task_id

Using Fixtures

Common fixtures are defined in tests/python/shared/fixtures/:

@pytest.fixture
def admin_user():
    """Fixture providing an authenticated admin user."""
    # Returns a test client authenticated as admin
    pass

@pytest.fixture
def tasks():
    """Fixture providing test tasks."""
    # Returns list of test tasks
    pass

Writing Frontend Unit Tests

Frontend tests use Jest and React Testing Library.

Component Test Example

import { render, screen, fireEvent } from '@testing-library/react';
import TaskItem from './TaskItem';

describe('<TaskItem />', () => {
    const mockTask = {
        id: 1,
        name: 'Test Task',
        status: 'annotation',
    };
    
    const mockOnSelect = jest.fn();
    
    beforeEach(() => {
        mockOnSelect.mockClear();
    });
    
    it('renders task name', () => {
        render(<TaskItem task={mockTask} onSelect={mockOnSelect} />);
        expect(screen.getByText('Test Task')).toBeInTheDocument();
    });
    
    it('calls onSelect when clicked', () => {
        render(<TaskItem task={mockTask} onSelect={mockOnSelect} />);
        fireEvent.click(screen.getByText('Test Task'));
        expect(mockOnSelect).toHaveBeenCalledWith(1);
    });
    
    it('displays task status', () => {
        render(<TaskItem task={mockTask} onSelect={mockOnSelect} />);
        expect(screen.getByText('annotation')).toBeInTheDocument();
    });
});

Writing E2E Tests

E2E tests are in tests/cypress/e2e/.

Cypress Test Example

describe('Task creation', () => {
    beforeEach(() => {
        cy.visit('/');
        cy.login('admin1', 'password');
    });
    
    it('creates a new task', () => {
        // Navigate to task creation
        cy.contains('Create new task').click();
        
        // Fill in task details
        cy.get('[placeholder="Task name"]').type('My Test Task');
        cy.contains('Add label').click();
        cy.get('[placeholder="Label name"]').type('car');
        
        // Upload images
        cy.get('input[type="file"]').attachFile('sample_image.jpg');
        
        // Submit
        cy.contains('Submit').click();
        
        // Verify task was created
        cy.contains('Task has been created').should('be.visible');
        cy.contains('My Test Task').should('be.visible');
    });
});

Test Database

The test infrastructure uses a pre-populated database.

Database Reset

The database is automatically restored after each test function to ensure test isolation.

Updating Test Database

If you need to update the test database:

Start test services:
```
pytest tests/python/ --start-services
```
Make changes through the UI or API

Backup the database:

cd tests/python
docker exec test_cvat_server_1 python manage.py dumpdata --indent 2 --natural-foreign \
    --exclude=admin --exclude=auth.permission --exclude=authtoken --exclude=contenttypes \
    --exclude=django_rq --exclude=sessions \
    > shared/assets/cvat_db/data.json

Backup data volume:

docker exec test_cvat_server_1 tar --exclude "/home/django/data/cache" -cjv /home/django/data \
    > shared/assets/cvat_db/cvat_data.tar.bz2

Update JSON assets:
```
python shared/utils/dump_objects.py
```

Test Coverage

Check test coverage for Python:

pytest tests/python/ --cov=cvat --cov-report=html

View coverage report:

open htmlcov/index.html

For frontend coverage:

cd cvat-ui
yarn test --coverage

Debugging Tests

Python Tests

Add breakpoints:

import pdb; pdb.set_trace()

Run with debug output:

pytest tests/python/ -vv -s

Frontend Tests

Run in watch mode:

yarn test --watch

Use console.log or debugger statements.

Cypress Tests

Use interactive mode:

yarn run cypress:open

Add Cypress debug commands:

cy.debug();
cy.pause();

Continuous Integration

CVAT’s CI automatically runs:

All Python tests
Frontend unit tests
E2E tests
Linters and type checks
Security scans

Tests must pass before PRs can be merged.

Troubleshooting

Port Conflicts

If test containers fail to start due to port conflicts:

docker ps -a
docker stop $(docker ps -a -q)

Database Issues

Recreate the test database:

docker exec test_cvat_db_1 dropdb --if-exists cvat
docker exec test_cvat_db_1 createdb cvat
docker exec test_cvat_server_1 python manage.py migrate

Test Failures

For date/time related failures, update JSON assets:

python tests/python/shared/utils/dump_objects.py

Best Practices

Write isolated tests: Each test should be independent
Use descriptive names: Test names should clearly describe what they test
Follow AAA pattern: Arrange, Act, Assert
Test edge cases: Include tests for error conditions
Keep tests fast: Mock external dependencies when possible
Maintain test data: Keep test fixtures up to date
Clean up: Ensure tests don’t leave side effects

Development

Resources

​Test Structure

​Prerequisites

​Running Python Tests

​REST API Tests

​Run All Tests

​Run Specific Test Files

​Run Specific Test Functions

​Run with Verbose Output

​Run Tests in Parallel

​SDK Tests

​CLI Tests

​Keep Test Services Running

​Running Frontend Tests

​Unit Tests

​Running E2E Tests

​Interactive Mode

​Headless Mode

​E2E Test Configuration

​Writing Tests

​Writing Python REST API Tests

​Test Structure

​Using Fixtures

​Writing Frontend Unit Tests

​Component Test Example

​Writing E2E Tests

​Cypress Test Example

​Test Database

​Database Reset

​Updating Test Database

​Test Coverage

​Debugging Tests

​Python Tests

​Frontend Tests

​Cypress Tests

​Continuous Integration

​Troubleshooting

​Port Conflicts

​Database Issues

​Test Failures

​Best Practices

​Next Steps

Build docs developers (and LLMs) love

Test Structure

Prerequisites

Running Python Tests

REST API Tests

Run All Tests

Run Specific Test Files

Run Specific Test Functions

Run with Verbose Output

Run Tests in Parallel

SDK Tests

CLI Tests

Keep Test Services Running

Running Frontend Tests

Unit Tests

Running E2E Tests

Interactive Mode

Headless Mode

E2E Test Configuration

Writing Tests

Writing Python REST API Tests

Test Structure

Using Fixtures

Writing Frontend Unit Tests

Component Test Example

Writing E2E Tests

Cypress Test Example

Test Database

Database Reset

Updating Test Database

Test Coverage

Debugging Tests

Python Tests

Frontend Tests

Cypress Tests

Continuous Integration

Troubleshooting

Port Conflicts

Database Issues

Test Failures

Best Practices

Next Steps