Test Isolation

Test isolation is crucial for reliable, maintainable test suites. Patrol provides multiple mechanisms to ensure each test runs in a clean environment, independent of other tests. This page explains how isolation works and how to configure it for your needs.

Why Test Isolation Matters

Without proper isolation, tests can:

Fail intermittently due to state left by previous tests
Pass in isolation but fail in suite due to execution order dependencies
Produce false positives when relying on state from earlier tests
Be impossible to parallelize because they interfere with each other

Poor test isolation is one of the most common causes of flaky tests. Always design tests to be independent.

How Patrol Ensures Isolation

Patrol uses different strategies depending on the platform:

Android: Test Orchestrator

On Android, Patrol leverages AndroidX Test Orchestrator to run each test in a separate process.

Enable Test Orchestrator

In your android/app/build.gradle.kts (or build.gradle):

build.gradle.kts

android {
  defaultConfig {
    testInstrumentationRunner = "pl.leancode.patrol.PatrolJUnitRunner"
    testInstrumentationRunnerArguments["clearPackageData"] = "true"
  }
  
  testOptions {
    execution = "ANDROIDX_TEST_ORCHESTRATOR"
  }
}

dependencies {
  androidTestUtil("androidx.test:orchestrator:1.5.1")
}

How it works

With orchestrator enabled:

Before each test: Android installs and launches your app
During test: Test executes in isolated process
After test: App is force-stopped and data cleared
Next test: Fresh app installation and launch

This ensures complete isolation - no state persists between tests.

The clearPackageData flag is critical. It instructs the orchestrator to clear all app data (SharedPreferences, databases, files, etc.) between tests.

iOS: Full Isolation Flag

On iOS Simulator, Patrol provides the --full-isolation flag:

patrol test --full-isolation

This flag enables per-test isolation:

Before each test

Simulator is cloned to a fresh instance

During test

Test runs on the cloned simulator

After test

Cloned simulator is deleted

Next test

Process repeats with a new clone

iOS Full Isolation is ExperimentalThe --full-isolation flag for iOS is experimental and may be removed or changed in future releases. Use it with caution in production CI/CD pipelines.It’s also only available on iOS Simulator, not on physical devices.

Web: Natural Isolation

Flutter web tests run in isolated browser contexts via Playwright, providing natural isolation:

Each test runs in a fresh browser context
Local storage is cleared between tests
Cookies are isolated per context
No manual configuration needed

Test Bundle Architecture

Patrol’s test bundling system contributes to isolation:

How Test Bundling Works

Test Discovery

Patrol scans your test directory (default: patrol_test/) for files ending in _test.dart

Bundle Generation

Creates test_bundle.dart containing all test references:

test_bundle.dart (generated)

// This file is generated by Patrol CLI
import 'login_test.dart' as login_test;
import 'signup_test.dart' as signup_test;
import 'profile_test.dart' as profile_test;

void main() {
  login_test.main();
  signup_test.main();
  profile_test.main();
}

Native Test Setup

Native instrumentation (Android JUnit / iOS XCTest) lists all tests and runs them individually

Isolated Execution

Each test runs in isolation per platform configuration

Add to .gitignore:

patrol_test/test_bundle.dart

This file is auto-generated and should not be committed.

Sharding for Parallel Execution

Sharding allows you to split your test suite across multiple devices or machines for faster execution.

Web Sharding

Patrol supports sharding for web tests:

# Run 1st shard of 3 total
patrol test --device chrome --web-shard=1/3

# Run 2nd shard of 3 total  
patrol test --device chrome --web-shard=2/3

# Run 3rd shard of 3 total
patrol test --device chrome --web-shard=3/3

Mobile Sharding Strategy

While Patrol doesn’t have built-in mobile sharding, you can implement it manually:

# Run first half of tests
patrol test \
  --target test1.dart \
  --target test2.dart \
  --target test3.dart

# Run second half of tests
patrol test \
  --target test4.dart \
  --target test5.dart \
  --target test6.dart

Test Configuration Directory

Customize test directory location in pubspec.yaml:

pubspec.yaml

patrol:
  app_name: My App
  test_directory: integration_test  # Default is patrol_test
  android:
    package_name: com.example.myapp
  ios:
    bundle_id: com.example.MyApp

Using a custom test directory is useful when:

Migrating from integration_test package
Following existing project structure conventions
Organizing different types of tests in separate directories

Writing Isolated Tests

Even with perfect platform isolation, your tests should be designed for independence:

Initialize Fresh State

patrolTest('user can login', ($) async {
  // Start with explicit state
  await clearUserSession();
  await $.pumpWidgetAndSettle(MyApp());
  
  // Test knows it's starting clean
  expect($(LoginScreen).exists, equals(true));
  
  await $(#emailField).enterText('[email protected]');
  await $(#passwordField).enterText('password');
  await $(#loginButton).tap();
  
  await $(Dashboard).waitUntilVisible();
});

Avoid Global State

int testCounter = 0;  // Shared across tests!

patrolTest('test 1', ($) async {
  testCounter++;
  expect(testCounter, 1);  // Fails if tests run out of order
});

patrolTest('test 2', ($) async {
  testCounter++;
  expect(testCounter, 2);  // Order-dependent
});

Clean Up Resources

Resource Cleanup

patrolTest('test with resources', ($) async {
  final server = await startMockServer();
  final subscription = eventStream.listen((_) {});
  
  try {
    await $.pumpWidgetAndSettle(MyApp());
    
    // Test code
    await $(#dataButton).tap();
    await $(#result).waitUntilVisible();
  } finally {
    // Always clean up
    await server.close();
    await subscription.cancel();
  }
});

Isolation in CI/CD

Firebase Test Lab

Firebase Test Lab provides excellent isolation:

Each test runs on a fresh device/emulator
Devices are wiped between test runs
Perfect for Android clearPackageData setup

.github/workflows/test.yml

- name: Build for Firebase Test Lab
  run: patrol build android
  
- name: Run on Firebase Test Lab
  run: |
    gcloud firebase test android run \
      --type instrumentation \
      --app build/app/outputs/apk/debug/app-debug.apk \
      --test build/app/outputs/apk/androidTest/debug/app-debug-androidTest.apk \
      --device model=Pixel2,version=30

GitHub Actions

Run tests with isolation on GitHub Actions:

name: Android Tests

on: [push, pull_request]

jobs:
  test:
    runs-on: macos-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup Flutter
        uses: subosito/flutter-action@v2
        
      - name: Install Patrol CLI
        run: flutter pub global activate patrol_cli
        
      - name: Run tests
        uses: reactivecircus/android-emulator-runner@v2
        with:
          api-level: 30
          script: patrol test

Debugging Isolation Issues

Test Passes Alone But Fails in Suite

Run test in isolation

patrol test --target patrol_test/problem_test.dart

If it passes, there’s an isolation issue.

Run with previous test

patrol test \
  --target patrol_test/test_before.dart \
  --target patrol_test/problem_test.dart

Does it fail now? test_before.dart is affecting state.

Identify state pollution

Look for:

Global variables
Singleton services
Persistent storage
Network state
Permission grants

Fix the issue

Add cleanup in problem_test.dart or fix state management in test_before.dart.

Check Isolation Configuration

Android isolation not working

Verify your build.gradle has:

testInstrumentationRunnerArguments["clearPackageData"] = "true"

testOptions {
  execution = "ANDROIDX_TEST_ORCHESTRATOR"
}

androidTestUtil("androidx.test:orchestrator:1.5.1")

Check logcat for orchestrator messages:

adb logcat | grep Orchestrator

iOS isolation not working

Make sure you’re using the flag:

patrol test --full-isolation

Check for simulator cloning in logs. If physical device, isolation isn’t supported.

Best Practices

Test Isolation Checklist:✅ Enable clearPackageData on Android✅ Use --full-isolation on iOS Simulator (when needed)✅ Initialize fresh state at test start✅ Avoid global mutable state✅ Clean up resources in finally blocks✅ Don’t rely on test execution order✅ Make tests idempotent - safe to run multiple times✅ Add test_bundle.dart to .gitignore✅ Test in isolation AND in full suite✅ Consider sharding for large test suites

Common Pitfalls

Avoid these anti-patterns:

Sequential test dependencies

// BAD: test2 depends on test1
patrolTest('test1 - create account', ($) async { ... });
patrolTest('test2 - login with account', ($) async { ... });

Shared test fixtures without cleanup

// BAD: File created by test1 affects test2
final testFile = File('test_data.json');

Assuming app state

// BAD: Assumes clean SharedPreferences
final prefs = await SharedPreferences.getInstance();
expect(prefs.getString('key'), isNull);

Not waiting for async operations

// BAD: State might not be ready
await deleteAllData();  // async
await $.pumpWidgetAndSettle();  // Might render old data

Getting Started

Core Concepts

Custom Finders

Native Automation

Platform Support

Test Isolation

Why Test Isolation Matters

How Patrol Ensures Isolation

Android: Test Orchestrator

iOS: Full Isolation Flag

Web: Natural Isolation

Test Bundle Architecture

How Test Bundling Works

Sharding for Parallel Execution

Web Sharding

Mobile Sharding Strategy

Test Configuration Directory

Writing Isolated Tests

Initialize Fresh State

Avoid Global State

Clean Up Resources

Isolation in CI/CD

Firebase Test Lab

GitHub Actions

Debugging Isolation Issues

Test Passes Alone But Fails in Suite

Check Isolation Configuration

Best Practices

Common Pitfalls

Next Steps

CI/CD Setup

Best Practices

Build docs developers (and LLMs) love

Getting Started

Core Concepts

Custom Finders

Native Automation

Platform Support

​Why Test Isolation Matters

​How Patrol Ensures Isolation

​Android: Test Orchestrator

​iOS: Full Isolation Flag

​Web: Natural Isolation

​Test Bundle Architecture

​How Test Bundling Works

​Sharding for Parallel Execution

​Web Sharding

​Mobile Sharding Strategy

​Test Configuration Directory

​Writing Isolated Tests

​Initialize Fresh State

​Avoid Global State

​Clean Up Resources

​Isolation in CI/CD

​Firebase Test Lab

​GitHub Actions

​Debugging Isolation Issues

​Test Passes Alone But Fails in Suite

​Check Isolation Configuration

​Best Practices

​Common Pitfalls

​Next Steps

CI/CD Setup

Best Practices

Build docs developers (and LLMs) love

Why Test Isolation Matters

How Patrol Ensures Isolation

Android: Test Orchestrator

iOS: Full Isolation Flag

Web: Natural Isolation

Test Bundle Architecture

How Test Bundling Works

Sharding for Parallel Execution

Web Sharding

Mobile Sharding Strategy

Test Configuration Directory

Writing Isolated Tests

Initialize Fresh State

Avoid Global State

Clean Up Resources

Isolation in CI/CD

Firebase Test Lab

GitHub Actions

Debugging Isolation Issues

Test Passes Alone But Fails in Suite

Check Isolation Configuration

Best Practices

Common Pitfalls

Next Steps