Testing TypeScript

Running Tests

Quick Test Run

Run all tests in parallel (recommended):

hereby runtests-parallel

Or use the npm script:

npm test

This builds the test infrastructure and runs all test suites using multiple worker threads.

By default, hereby uses one worker per CPU core. Adjust with --workers=<number>.

Serial Test Execution

Run tests sequentially for debugging:

hereby runtests

Test Suites

TypeScript has several test suites organized by type:

Compiler Tests
Conformance Tests
FourSlash Tests
Project Tests

Tests in tests/cases/compiler/ verify compiler behavior:

hereby runtests --tests=compiler

These test compilation, emit, and error detection.

Tests in tests/cases/conformance/ verify spec compliance:

hereby runtests --tests=conformance

Organized by spec section (types, expressions, statements, etc.).

Tests in tests/cases/fourslash/ test language service features:

hereby runtests --tests=fourslash

Includes completions, refactorings, navigation, and more.

Tests in tests/cases/projects/ test multi-file scenarios:

hereby runtests --tests=project

Tests project references and build mode.

Running Specific Tests

By Name Pattern

Run tests matching a regex:

hereby runtests --tests=<pattern>

Examples:

# Run tests with "async" in the name
hereby runtests --tests=async

# Run a specific test file
hereby runtests --tests=2dArrays

# Run tests in a specific suite
hereby runtests --tests=conformance/types

By Runner

Specify which test runner to use:

hereby runtests --runner=<runnerName>

Available runners:

conformance - Conformance suite
compiler - Compiler suite
fourslash - Language service suite
project - Project suite

From Failed Tests File

Rerun previously failed tests:

hereby runtests --failed

Failed tests are tracked in .failed-tests.

Test Options

Common Options
Advanced Options
Reporters

# Light mode (faster, fewer verifications)
hereby runtests-parallel --light

# Keep failed tests in .failed-tests even if they pass
hereby runtests --keepFailed

# Don't clean test output directories first
hereby runtests --dirty

# Custom timeout (default: 40000ms)
hereby runtests --timeout=60000

# Disable colors
hereby runtests --no-color

# Custom stack trace limit
hereby runtests --stackTraceLimit=50
hereby runtests --stackTraceLimit=full

# Generate code coverage with c8
hereby runtests-parallel --coverage

# Run on a specific shard (for CI)
hereby runtests-parallel --shards=4 --shardId=1

# Use built compiler instead of LKG
hereby runtests --built

# Specify number of parallel workers
hereby runtests-parallel --workers=8

# Default parallel reporter (minimal output)
hereby runtests-parallel --reporter=min

# Five-line format (serial runs)
hereby runtests --reporter=mocha-fivemat-progress-reporter

# Mocha's spec reporter
hereby runtests --reporter=spec

# Mocha's dot reporter
hereby runtests --reporter=dot

Watch Mode

Automatically rerun tests when files change:

hereby runtests-watch --tests=<pattern>

You must specify --tests or --failed to use watch mode.

Example:

hereby runtests-watch --tests=asyncAwait

This watches:

Test infrastructure (built/local/run.js)
Test cases (tests/cases/**)
Test libraries (tests/lib/**)

Debugging Tests

Inspector Mode

Run tests with Node.js inspector:

hereby runtests --tests=<pattern> -i

Example:

hereby runtests --tests=2dArrays -i

Then open chrome://inspect in Chrome or attach from VS Code.

VS Code Debugging

Create Launch Configuration

Copy .vscode/launch.template.json to .vscode/launch.json:

cp .vscode/launch.template.json .vscode/launch.json

Open Test File

Open the .ts test file you want to debug in VS Code.

Launch Debugger

Press F5 or select Run > Start Debugging.The configuration runs Mocha with the currently open test file name.

Set Breakpoints

Set breakpoints in either:

Test files (tests/cases/**)
Compiler source (src/**)

The debugger uses source maps, so breakpoints in TypeScript source work directly.

Test Baselines

Most tests generate baseline files to detect output changes:

Baseline Files

Compiler tests generate:

.js - Emitted JavaScript
.d.ts - Emitted declarations (in .js file)
.errors.txt - Compiler errors
.types - Type of each expression
.symbols - Symbol for each identifier
.js.map - Source maps (if requested)

Comparing Baselines

After running tests, compare changes:

hereby diff

Or manually with git:

git diff --no-index tests/baselines/reference tests/baselines/local

Set the DIFF environment variable to use your preferred diff tool:

export DIFF="code --diff"
hereby diff

Accepting Baselines

If the baseline changes are correct:

hereby baseline-accept

This copies tests/baselines/local/* to tests/baselines/reference/.

Carefully review baseline changes before accepting! Unexpected changes may indicate bugs.

Writing New Tests

Adding a Compiler Test

Create Test File

Add a .ts file in tests/cases/compiler/:

tests/cases/compiler/myNewFeature.ts

// @strict: true
// @target: es2020

// Test code that demonstrates the bug fix or new feature
function example() {
    const x: string = "hello";
    return x.toUpperCase();
}

Add Metadata

Use comment tags to configure test behavior:

// @strict: true              // Enable strict mode
// @target: es2020            // Set compilation target
// @module: commonjs          // Set module system
// @noEmit: true              // Don't emit output
// @noImplicitAny: true       // Specific compiler option

Run the Test

hereby runtests --tests=myNewFeature

Accept Baselines

hereby baseline-accept

Multi-File Tests

Use @filename tags to simulate multiple files:

// @filename: types.ts
export interface User {
    name: string;
    age: number;
}

// @filename: main.ts
import { User } from "./types";

const user: User = {
    name: "Alice",
    age: 30
};

Conformance Tests

For spec compliance tests, add files to appropriate subdirectories:

tests/cases/conformance/
├── types/               # Type system tests
├── expressions/         # Expression tests
├── statements/          # Statement tests
├── es6/                 # ES6 feature tests
├── es2015/              # ES2015 tests
└── ...

Conformance test names must be unique across all test directories.

Test Infrastructure

The test runner is built from src/testRunner/:

# Build test infrastructure
hereby tests

# Or use npm script
npm run build:tests

Output: built/local/run.js

Test Harness

The harness code in src/harness/ provides utilities:

Compiler host implementations
Virtual file systems
Baseline management
Test configuration parsing

ESLint Rule Tests

TypeScript includes custom ESLint rules with their own tests:

hereby run-eslint-rules-tests

Or:

npm run test:eslint-rules

These tests are separate from the main compiler test suite.

Continuous Integration

Sharded Testing

CI systems can split tests across multiple machines:

# Run shard 1 of 4
hereby runtests-parallel --shards=4 --shardId=1

# Run shard 2 of 4
hereby runtests-parallel --shards=4 --shardId=2

Coverage Reports

Generate test coverage with c8:

hereby runtests-parallel --coverage

Coverage reports are written to the coverage/ directory.

Troubleshooting

Tests fail with 'Cannot find module'

Build the test infrastructure:

hereby tests

Baseline comparison shows no differences but test fails

Regenerate baselines:

hereby runtests --tests=<test> --dirty
hereby baseline-accept

Test times out

Increase timeout:

hereby runtests --tests=<test> --timeout=120000

Watch mode doesn't detect changes

Watch mode is experimental. Use manual reruns:

hereby runtests --tests=<pattern>

Debugger doesn't hit breakpoints

Ensure you’ve built tests with source maps:

hereby tests

Source maps are enabled by default in Herebyfile.mjs.

Test Guidelines

Test Requirements

Every PR should include:

At least one test that fails without your changes
Reasonable permutations of the fix/feature
Updated baselines if output changes
Test names that clearly describe what they test

Next Steps

Debugging

Learn debugging techniques

Building

Build system reference

Development

Architecture

Running Tests

Quick Test Run

Serial Test Execution

Test Suites

Running Specific Tests

By Name Pattern

By Runner

From Failed Tests File

Test Options

Watch Mode

Debugging Tests

Inspector Mode

VS Code Debugging

Test Baselines

Baseline Files

Comparing Baselines

Accepting Baselines

Writing New Tests

Adding a Compiler Test

Multi-File Tests

Conformance Tests

Test Infrastructure

Test Harness

ESLint Rule Tests

Continuous Integration

Sharded Testing

Coverage Reports

Troubleshooting

Test Guidelines

Test Requirements

Next Steps

Debugging

Building

Build docs developers (and LLMs) love

Development

Architecture

​Running Tests

​Quick Test Run

​Serial Test Execution

​Test Suites

​Running Specific Tests

​By Name Pattern

​By Runner

​From Failed Tests File

​Test Options

​Watch Mode

​Debugging Tests

​Inspector Mode

​VS Code Debugging

​Test Baselines

​Baseline Files

​Comparing Baselines

​Accepting Baselines

​Writing New Tests

​Adding a Compiler Test

​Multi-File Tests

​Conformance Tests

​Test Infrastructure

​Test Harness

​ESLint Rule Tests

​Continuous Integration

​Sharded Testing

​Coverage Reports

​Troubleshooting

​Test Guidelines

Test Requirements

​Next Steps

Debugging

Building

Build docs developers (and LLMs) love

Running Tests

Quick Test Run

Serial Test Execution

Test Suites

Running Specific Tests

By Name Pattern

By Runner

From Failed Tests File

Test Options

Watch Mode

Debugging Tests

Inspector Mode

VS Code Debugging

Test Baselines

Baseline Files

Comparing Baselines

Accepting Baselines

Writing New Tests

Adding a Compiler Test

Multi-File Tests

Conformance Tests

Test Infrastructure

Test Harness

ESLint Rule Tests

Continuous Integration

Sharded Testing

Coverage Reports

Troubleshooting

Test Guidelines

Next Steps