Validation and Testing Tools

This page documents the validation and testing tools used to ensure the integrity of Helium’s build system and configuration.

Overview

Helium includes comprehensive validation tools to catch issues early in the development process. These tools verify patch compatibility, configuration correctness, and build consistency.

Automated Validation

validate_patches.py - Complete Patch Validation

The most comprehensive validation tool, checking that all patches apply cleanly to the Chromium source tree.

Load patches

Reads all patches from the series file and parses them as unified diffs

Resolve dependencies

Parses Chromium’s DEPS file to understand the repository structure and dependency versions

Retrieve source files

Downloads required files from Google’s repositories or reads from local source tree

Apply patches

Applies each patch to an in-memory representation of the source tree

Report results

Logs any patches that fail to apply cleanly or have validation errors

python3 devutils/validate_patches.py \
  --remote \
  --verbose

When to use which mode:

Remote: CI pipelines, automated testing, clean validation
Local: Fast development iteration when you have source tree
Cache: Debugging patch issues, inspecting source files

validate_config.py - Configuration Integrity

All-in-one validation for Helium’s configuration files.

What it validates
Usage
CI Integration

All patches exist at specified paths

All patches are referenced in series file

No duplicate patches in series

Each patch is parseable as unified diff

GN flags are alphabetically sorted

No duplicate GN flag definitions

downloads.ini conforms to schema

Download URLs and hashes are valid

# Run from repository root
python3 devutils/validate_config.py

# Returns 0 on success, 1 on any warnings/errors
echo $?

# .github/workflows/validate.yml
- name: Validate configuration
  run: python3 devutils/validate_config.py
  
- name: Validate patches
  run: |
    python3 devutils/validate_patches.py \
      --remote \
      --verbose

Individual Validators

Patch File Validation

# Validates patch file structure and series
python3 devutils/check_patch_files.py

Validation checks performed

Readability: All patches can be opened and read
Parseability: Each patch is valid unified diff format
Completeness: All referenced patches exist on disk
Coverage: No unused patches in directory
Uniqueness: No duplicate patch references

GN Flags Validation

# Validates GN build flags
python3 devutils/check_gn_flags.py

Common issues caught:

Unsorted flags (affects diff readability)
Duplicate flag definitions (causes build errors)
Malformed flag syntax

GN flags must be sorted alphabetically. This ensures consistent diffs and makes it easier to track changes across versions.

Downloads Configuration

# Validates download configurations
python3 devutils/check_downloads_ini.py

Schema validation

Verifies:

Required fields present (url, download_filename, output_path)
Hash specifications valid (md5, sha1, sha256, sha512)
Extractor types supported (tar, 7z, winrar)
Output paths are relative
URL variables are valid ()
Hash URL format correct (scheme|path|algorithm)

File Existence Check

# Quick validation for file lists
python3 devutils/check_files_exist.py \
  /path/to/chromium/src \
  pruning.list

Useful for CI checks to quickly verify that:

Pruning lists reference valid files
Domain substitution targets exist
Resource files are present
Test data is available

Testing Workflows

Pre-Commit Hook

Add to .git/hooks/pre-commit:

#!/bin/bash
set -e

echo "Validating configuration..."
python3 devutils/validate_config.py

echo "Running code quality checks..."  
python3 devutils/run_devutils_pylint.py
python3 devutils/run_utils_pylint.py

echo "All validations passed!"

CI Pipeline

GitHub Actions
GitLab CI
Local Testing

name: Validate
on: [push, pull_request]

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'
          
      - name: Install dependencies
        run: pip install requests
        
      - name: Validate configuration
        run: python3 devutils/validate_config.py
        
      - name: Validate patches (remote)
        run: |
          python3 devutils/validate_patches.py \
            --remote \
            --verbose
            
      - name: Code quality
        run: |
          pip install pylint
          python3 devutils/run_devutils_pylint.py
          python3 devutils/run_utils_pylint.py

validate:config:
  stage: validate
  script:
    - python3 devutils/validate_config.py
    
validate:patches:
  stage: validate
  script:
    - pip install requests
    - python3 devutils/validate_patches.py --remote --verbose
    
validate:quality:
  stage: validate
  script:
    - pip install pylint
    - python3 devutils/run_devutils_pylint.py
    - python3 devutils/run_utils_pylint.py

#!/bin/bash
# test-all.sh - Run all validations locally

set -e

echo "==> Validating configuration"
python3 devutils/validate_config.py

echo "==> Validating patches against local source"
if [ -d "../chromium/src" ]; then
  python3 devutils/validate_patches.py \
    --local ../chromium/src
else
  echo "Skipping: no local source tree found"
fi

echo "==> Running code quality checks"
if command -v pylint &> /dev/null; then
  python3 devutils/run_devutils_pylint.py
  python3 devutils/run_utils_pylint.py
else
  echo "Skipping: pylint not installed"
fi

echo "✓ All validations passed!"

Development Iteration

Make changes

Modify patches, configuration, or build scripts

Quick validation

python3 devutils/validate_config.py

Patch validation

# If you have local source tree
python3 devutils/validate_patches.py \
  --local /path/to/chromium/src

Full validation

# Before committing
python3 devutils/validate_patches.py --remote

Debugging Failed Validations

Patch Application Failures

Patch does not apply cleanly

Problem: Patch fails with context mismatchSolutions:

Update patch for current Chromium version:

# Apply what you can and regenerate
cd /path/to/chromium/src
patch -p1 < /path/to/patch.patch
# Make manual fixes
git diff > /path/to/updated_patch.patch

Check if file was moved or renamed:

# Search for file content in source tree
cd /path/to/chromium/src
grep -r "unique_pattern_from_patch" .

Use verbose mode to see exact failure:

python3 devutils/validate_patches.py \
  --local /path/to/chromium/src \
  --verbose

File not found in source tree

Problem: Patch references file that doesn’t existCauses:

File was removed in newer Chromium version
File path changed
File is in a dependency repository

Debug:

# Check if file exists anywhere
find /path/to/chromium/src -name "filename.cc"

# Check git history
cd /path/to/chromium/src
git log --all --full-history -- path/to/file.cc

Hash mismatch after domain substitution

Problem: Revert fails due to hash mismatchCauses:

Files modified after domain substitution
Cache doesn’t match current source tree

Solutions:

Don’t modify files after domain substitution

If needed, revert before making changes:

python3 utils/domain_substitution.py revert \
  --cache domainsub.tar.gz \
  /path/to/src
# Make your changes
python3 utils/domain_substitution.py apply \
  --regex domain_regex.list \
  --files domain_substitution.list \
  --cache domainsub.tar.gz \
  /path/to/src

Configuration Issues

GN flags out of order

Problem: check_gn_flags.py reports unsorted flagsSolution:

# Sort flags alphabetically
sort -o flags.gn flags.gn

# Verify
python3 devutils/check_gn_flags.py

Duplicate patches in series

Problem: Same patch listed multiple timesSolution:

Edit patches/series
Remove duplicate lines
Verify:
```
python3 devutils/check_patch_files.py
```

Unused patches detected

Problem: Patch files exist but aren’t in seriesDecide:

Add to patches/series if needed
Delete if obsolete
Move to archive directory if keeping for reference

Verify:

# Check what's unused
python3 devutils/check_patch_files.py

# Add to series or delete
rm patches/path/to/unused.patch

Validation in Build Scripts

Integrate validation into build scripts:

#!/bin/bash
# build.sh - Build with validation

set -e

# Validate before building
echo "==> Validating configuration"
python3 devutils/validate_config.py

# Validate source tree after preparation
echo "==> Validating source tree"
python3 devutils/check_files_exist.py \
  "$SOURCE_DIR" \
  pruning.list \
  domain_substitution.list

# Build
echo "==> Building"
gn gen out/Release
ninja -C out/Release chrome

echo "✓ Build complete"

Performance Considerations

validate_patches.py --remote

Time: 10-20 minutes
Use: CI, clean validation
Requires: Internet, requests module

validate_patches.py --local

Time: 1-2 minutes
Use: Development, iteration
Requires: Local source tree

validate_config.py

Time: 5-10 seconds
Use: Quick checks, pre-commit
Requires: Config files only

check_files_exist.py

Time: 1-5 seconds
Use: CI, quick validation
Requires: Source tree, file lists

Exit Codes Reference

All validation scripts use consistent exit codes:

integer

Success - All validations passed

integer

Failure - One or more validations failed or warnings occurred

Use in scripts:

if python3 devutils/validate_config.py; then
  echo "Configuration valid"
else
  echo "Configuration has errors" >&2
  exit 1
fi

Best Practices

Always validate before committing

Run at minimum:

python3 devutils/validate_config.py

For patch changes:

python3 devutils/validate_patches.py --local /path/to/src

Use CI for comprehensive checks

Local validation is fast but may miss issues. Always have CI run:

validate_config.py
validate_patches.py --remote
Code quality checks (pylint)

Cache source files for debugging

When debugging patch issues:

python3 devutils/validate_patches.py \
  --cache-remote /tmp/chromium_files

# Inspect cached files
ls -la /tmp/chromium_files/

Keep validation fast

Use --local during development
Only use --remote for final validation
Run validate_config.py frequently (it’s fast)

Developer Utilities

Complete reference for all developer utilities

Build Utilities

Core build utilities for patches and domain substitution

CI/CD Setup

Setting up automated builds and testing

Contributing

Guidelines for contributing to Helium

Building

Contributing

Utilities

Validation and Testing Tools

Overview

Automated Validation

validate_patches.py - Complete Patch Validation

validate_config.py - Configuration Integrity

Individual Validators

Patch File Validation

GN Flags Validation

Downloads Configuration

File Existence Check

Testing Workflows

Pre-Commit Hook

CI Pipeline

Development Iteration

Debugging Failed Validations

Patch Application Failures

Configuration Issues

Validation in Build Scripts

Performance Considerations

validate_patches.py --remote

validate_patches.py --local

validate_config.py

check_files_exist.py

Exit Codes Reference

Best Practices

See Also

Developer Utilities

Build Utilities

CI/CD Setup

Contributing

Build docs developers (and LLMs) love

Building

Contributing

Utilities

​Overview

​Automated Validation

​validate_patches.py - Complete Patch Validation

​validate_config.py - Configuration Integrity

​Individual Validators

​Patch File Validation

​GN Flags Validation

​Downloads Configuration

​File Existence Check

​Testing Workflows

​Pre-Commit Hook

​CI Pipeline

​Development Iteration

​Debugging Failed Validations

​Patch Application Failures

​Configuration Issues

​Validation in Build Scripts

​Performance Considerations

validate_patches.py --remote

validate_patches.py --local

validate_config.py

check_files_exist.py

​Exit Codes Reference

​Best Practices

​See Also

Developer Utilities

Build Utilities

CI/CD Setup

Contributing

Build docs developers (and LLMs) love

Overview

Automated Validation

validate_patches.py - Complete Patch Validation

validate_config.py - Configuration Integrity

Individual Validators

Patch File Validation

GN Flags Validation

Downloads Configuration

File Existence Check

Testing Workflows

Pre-Commit Hook

CI Pipeline

Development Iteration

Debugging Failed Validations

Patch Application Failures

Configuration Issues

Validation in Build Scripts

Performance Considerations

Exit Codes Reference

Best Practices

See Also