Testing Guidelines

Testing is a critical part of the Rust compiler development process. This guide covers the comprehensive testing infrastructure used in the project.

For detailed information about the compiler testing framework, see the rustc-dev-guide testing section.

Test Suite Overview

The Rust compiler project includes multiple test suites, each serving different purposes:

Test Suite	Purpose	Location
ui	Compile-fail and run-pass tests with error output verification	`tests/ui/`
run-make	Complex tests using Makefiles	`tests/run-make/`
codegen	LLVM IR generation tests	`tests/codegen-llvm/`
assembly	Assembly output tests	`tests/assembly-llvm/`
mir-opt	MIR optimization tests	`tests/mir-opt/`
debuginfo	Debug information tests	`tests/debuginfo/`
rustdoc	Documentation generation tests	`tests/rustdoc-html/`, `tests/rustdoc-ui/`
incremental	Incremental compilation tests	`tests/incremental/`
coverage	Code coverage tests	`tests/coverage/`

Running Tests

Basic Test Commands

All Tests
Specific Suite
Single Test
Fast Check

Run the entire test suite (takes several hours):

./x.py test

Running all tests takes significant time and resources. Use more targeted approaches during development.

Run a specific test suite:

# UI tests (most common)
./x.py test tests/ui

# Run-make tests
./x.py test tests/run-make

# MIR optimization tests
./x.py test tests/mir-opt

Run a specific test file:

# Run a specific UI test
./x.py test tests/ui/issues/issue-12345.rs

# Run tests matching a pattern
./x.py test tests/ui --test-args pattern

Quick checks without running the full test suite:

# Check if code compiles
./x.py check

# Run tidy (formatting and style checks)
./x.py test tidy

Test Flags and Options

# Run with specific number of threads
./x.py test tests/ui --test-args --test-threads=4

Writing UI Tests

UI tests are the most common type of test. They verify compiler output, including error messages.

Basic UI Test Structure

Create a test file in tests/ui/:

// Test for issue #12345: ICE with associated types in trait bounds

trait MyTrait {
    type Assoc;
}

fn foo<T: MyTrait<Assoc = i32>>() {
    // This should compile without ICE
}

fn main() {}

Test Directives

Use special comments to control test behavior:

Compilation Behavior

// compile-flags: -O -C no-prepopulate-passes
// check-pass
// build-pass
// run-pass
// compile-fail

Error Checking

fn main() {
    let x: i32 = "string"; //~ ERROR mismatched types
    //~| expected `i32`, found `&str`
}

The //~ markers indicate expected error locations:

//~ - Error on the same line
//~^ - Error on the previous line
//~| - Additional error message

Target Selection

// only-x86_64
// only-windows
// ignore-emscripten
// ignore-windows

Edition Specification

// edition:2021
// edition:2018

Updating Test Expectations

When error messages change, update expectations with “bless”:

./x.py test tests/ui/issues/issue-12345.rs --bless

This updates the .stderr or .stdout files to match the new output.

Always review blessed changes carefully to ensure they’re intentional improvements, not regressions.

Writing Run-Make Tests

For complex scenarios requiring multiple files or build steps:

Create Test Directory

mkdir tests/run-make/my-feature-test

Write Makefile

Create tests/run-make/my-feature-test/Makefile:

include ../tools.mk

all:
	$(RUSTC) --crate-type lib lib.rs
	$(RUSTC) --extern mylib=liblib.rlib main.rs
	$(call RUN,main)

Add Source Files

Create the necessary Rust source files in the same directory

Run the Test

./x.py test tests/run-make/my-feature-test

Codegen and Assembly Tests

Codegen Tests

Verify the LLVM IR generated by rustc:

// compile-flags: -O -C no-prepopulate-passes

#![crate_type = "lib"]

// CHECK-LABEL: @add_one
// CHECK: add i32 %{{.*}}, 1
// CHECK-NEXT: ret i32
#[no_mangle]
pub fn add_one(x: i32) -> i32 {
    x + 1
}

Assembly Tests

Verify the generated assembly code:

// assembly-output: emit-asm
// compile-flags: -O
// only-x86_64

#![crate_type = "lib"]

// CHECK-LABEL: multiply:
// CHECK: imul
#[no_mangle]
pub fn multiply(a: i32, b: i32) -> i32 {
    a * b
}

MIR Optimization Tests

Test the Middle Intermediate Representation (MIR) optimizations:

// EMIT_MIR const-prop.main.ConstProp.diff

fn main() {
    let x = 2 + 2;
    let y = x * 2;
}

Run and bless:

./x.py test tests/mir-opt/const-prop.rs --bless

Debugging Test Failures

Understanding Test Output

When a test fails, you’ll see:

failures:

---- [ui] tests/ui/issues/issue-12345.rs stdout ----

error: test compilation failed although it shouldn't!
status: exit code: 101
command: "/path/to/rustc" "tests/ui/issues/issue-12345.rs" ...
--stderr-------------------------------
error[E0308]: mismatched types
  --> tests/ui/issues/issue-12345.rs:10:18
   |
10 |     let x: i32 = "string";
   |            ---   ^^^^^^^^ expected `i32`, found `&str`

Common Debugging Strategies

Isolate the Issue
Check Compiler Output
Use Debug Prints
Compare Expected Output

Run just the failing test:

./x.py test tests/ui/issues/issue-12345.rs --verbose

Build the compiler and manually test:

./x.py build --stage 1
./build/x86_64-unknown-linux-gnu/stage1/bin/rustc tests/ui/issues/issue-12345.rs

Add debug output to the compiler:

eprintln!("Debug: value = {:?}", value);

Then run:

./x.py test tests/ui/issues/issue-12345.rs 2>&1 | grep "Debug:"

Look at the .stderr or .stdout file:

cat tests/ui/issues/issue-12345.stderr

Compare with actual output to see differences.

Incremental Testing

Test incremental compilation scenarios:

// revisions: rpass1 rpass2

#[cfg(rpass1)]
struct Foo {
    x: i32,
}

#[cfg(rpass2)]
struct Foo {
    x: i32,
    y: i32,  // Added field
}

fn main() {
    let _ = Foo { x: 0 };
}

The test runs in two passes to verify incremental behavior.

Coverage Testing

Test code coverage instrumentation:

// compile-flags: -C instrument-coverage

fn main() {
    covered_function();
}

fn covered_function() {
    println!("This should be covered");
}

fn uncovered_function() {
    println!("This should not be covered");
}

CI Testing Workflow

Local CI Simulation

Before pushing, simulate what CI will run:

Tidy Check

./x.py test tidy

Checks formatting, license headers, and project standards

UI Test Suite

./x.py test tests/ui

Most likely to catch issues

Build Tests

./x.py test --stage 1

Full build with stage 1 compiler

CI Job Matrix

The CI runs tests on multiple platforms defined in .github/workflows/ci.yml:

Linux: x86_64-gnu-llvm, x86_64-gnu-tools, i686-gnu
Windows: x86_64-msvc, i686-msvc, x86_64-gnu
macOS: x86_64-darwin, aarch64-darwin
Cross-compilation: Various embedded and tier 2 targets

You can see the job matrix definition in src/ci/github-actions/jobs.yml, which is processed by the citool to calculate which jobs run.

Performance Testing

For changes that might impact performance:

Request Performance Run

On your PR, comment:

@rust-timer queue

This queues a performance comparison against the master branch.

Interpret Results

The bot will report:

Instruction count changes
Wall-time changes
Memory usage changes
Binary size changes

Regression of more than 1-2% in instruction counts typically needs justification or mitigation.

Test Organization Best Practices

Name Tests Clearly

Use descriptive names like issue-12345.rs or trait-bound-ice.rs

Keep Tests Minimal

Reduce test cases to the smallest reproducing example

Add Comments

Explain what the test is checking and why

Link to Issues

Reference the issue or RFC in test comments

Use Correct Suite

Put tests in the appropriate suite (ui, run-make, etc.)

Update Expectations

Run with --bless when error messages change

Common Test Patterns

Regression Tests

For bug fixes, add a regression test:

// Regression test for issue #12345
// This used to cause an ICE

// check-pass

trait MyTrait {
    type Assoc;
}

fn foo<T: MyTrait<Assoc = i32>>() {}

fn main() {}

Feature Tests

For new features (behind feature gates):

// Test that my_feature requires a feature gate

fn main() {
    my_feature_syntax(); //~ ERROR my_feature is experimental
}

And the positive test:

// Test my_feature works when enabled

// check-pass
#![feature(my_feature)]

fn main() {
    my_feature_syntax();
}

Troubleshooting

Test Won’t Run

Stage Issues

Ensure you’re using the right stage:

# Build stage 1 first
./x.py build --stage 1

# Then test with stage 1
./x.py test --stage 1 tests/ui

Platform-Specific Tests

Some tests only run on specific platforms. Check the directives:

// only-x86_64
// only-linux

You may need to skip these locally.

Outdated Build Artifacts

Clean and rebuild:

rm -rf build/
./x.py build

Additional Resources

rustc-dev-guide: Tests

Comprehensive testing documentation

compiletest Documentation

Details on the test harness

Adding New Tests

Step-by-step guide for new tests

CI Configuration

View the CI workflow definition

Summary

Choose the Right Test Suite

Use UI tests for most cases, run-make for complex scenarios

Write Minimal Tests

Keep tests focused and as small as possible

Test Locally First

Run ./x.py test on your changes before pushing

Update Expectations

Use --bless when error messages legitimately change

Monitor CI

Watch CI results and fix failures promptly

Back to Workflow

Return to the contribution workflow guide

Contributing Guide

Development

​Test Suite Overview

​Running Tests

​Basic Test Commands

​Test Flags and Options

​Writing UI Tests

​Basic UI Test Structure

​Test Directives

​Updating Test Expectations

​Writing Run-Make Tests

​Codegen and Assembly Tests

​Codegen Tests

​Assembly Tests

​MIR Optimization Tests

​Debugging Test Failures

​Understanding Test Output

​Common Debugging Strategies

​Incremental Testing

​Coverage Testing

​CI Testing Workflow

​Local CI Simulation

​CI Job Matrix

​Performance Testing

​Request Performance Run

​Interpret Results

​Test Organization Best Practices

Name Tests Clearly

Keep Tests Minimal

Add Comments

Link to Issues

Use Correct Suite

Update Expectations

​Common Test Patterns

​Regression Tests

​Feature Tests

​Troubleshooting

​Test Won’t Run

​Additional Resources

rustc-dev-guide: Tests

compiletest Documentation

Adding New Tests

CI Configuration

​Summary

Back to Workflow

Build docs developers (and LLMs) love

Test Suite Overview

Running Tests

Basic Test Commands

Test Flags and Options

Writing UI Tests

Basic UI Test Structure

Test Directives

Updating Test Expectations

Writing Run-Make Tests

Codegen and Assembly Tests

Codegen Tests

Assembly Tests

MIR Optimization Tests

Debugging Test Failures

Understanding Test Output

Common Debugging Strategies

Incremental Testing

Coverage Testing

CI Testing Workflow

Local CI Simulation

CI Job Matrix

Performance Testing

Request Performance Run

Interpret Results

Test Organization Best Practices

Common Test Patterns

Regression Tests

Feature Tests

Troubleshooting

Test Won’t Run

Additional Resources

Summary