Skip to main content
Delta Sharing uses different testing frameworks for its Python and Scala/Java components. This guide covers running tests for all components of the project.

​
Python Connector Tests

The Python connector uses pytest for testing, including both unit tests and doctests.

​
Running All Python Tests

The simplest way to run all Python tests is using the provided test script: 
python/dev/pytest
This command runs:
  • Unit tests in the delta_sharing package
  • Doctests embedded in the module documentation
  • Integration tests with verbose output and colored results
The test script automatically ignores delta_sharing/_yarl_patch.py during automatic import to verify it’s loaded on-demand as expected.

​
Running Specific Test Files

To run tests from a specific file: 
python/dev/pytest delta_sharing/tests/test_delta_sharing.py

​
Running Specific Test Cases

Target individual test functions or classes: 
python/dev/pytest delta_sharing/tests/test_delta_sharing.py::test_profile_loading

​
Python Test Options

Customize test execution with pytest options: 
# Show detailed test information
python/dev/pytest -v

​
Python Test Environment

Tests can use environment variables for configuration: 
# Enable detailed logging in tests
export AWS_ACCESS_KEY_ID=your_test_key
python/dev/pytest
When AWS_ACCESS_KEY_ID is set, the test script enables verbose logging (-o log_cli=true -s).

​
Spark Connector and Server Tests

The Spark connector and Delta Sharing Server use ScalaTest via SBT.

​
Running All Scala Tests

Run the complete test suite for all Scala components: 
build/sbt test
This executes tests for:
  • Client library (delta-sharing-client)
  • Spark connector (delta-sharing-spark)
  • Delta Sharing Server (delta-sharing-server)

​
Running Tests for Specific Components

1

Test Client Only

build/sbt client/test
2

Test Spark Connector Only

build/sbt spark/test
3

Test Server Only

build/sbt server/test

​
Running Specific Test Suites

Execute individual test suites using the testOnly command: 
# Run a specific test suite
build/sbt "testOnly io.delta.sharing.client.DeltaSharingClientSuite"

# Run multiple test suites
build/sbt "testOnly io.delta.sharing.client.*Suite"

# Run tests matching a pattern
build/sbt "testOnly *DeltaSharing*"

​
Running Specific Tests Within a Suite

You can run individual tests by name: 
# Run a specific test by substring match
build/sbt "testOnly *DeltaSharingClientSuite -- -z 'list shares'"
The -z flag filters tests whose names contain the specified string.

​
Test Configuration

The build system includes optimized test configurations to improve performance and reduce memory usage.

​
JVM Test Options

Tests run with the following JVM settings (configured in build.sbt): 
Test / javaOptions ++= Seq(
  "-Dspark.ui.enabled=false",
  "-Dspark.ui.showConsoleProgress=false",
  "-Dspark.databricks.delta.snapshotPartitions=2",
  "-Dspark.sql.shuffle.partitions=5",
  "-Dspark.sql.sources.parallelPartitionDiscovery.parallelism=5",
  "-Dspark.delta.sharing.network.sslTrustAll=true",
  "-Xmx1024m"
)
These settings:
  • Disable Spark UI to reduce overhead
  • Limit partition counts for faster tests
  • Configure SSL for test environments
  • Set maximum heap size to 1GB

​
Azure Test Configuration

For tests requiring Azure access, configure credentials: 
export AZURE_TEST_ACCOUNT_KEY=your_account_key
build/sbt test

​
Cross-Version Testing

Test against multiple Scala versions: 
build/sbt ++2.12.18 test

​
Continuous Testing

Automatic test execution on file changes: 
# Re-run tests automatically when source files change
build/sbt ~test

# Watch specific project
build/sbt ~client/test
Press Enter to manually trigger a test run. Press Ctrl+C to exit watch mode.

​
Test Output and Debugging

​
Verbose Test Output

# Show full stack traces on failures
build/sbt "testOnly *Suite -- -oF"

​
Parallel Test Execution

By default, parallel execution is disabled (ThisBuild / parallelExecution := false in build.sbt) to prevent resource conflicts. To enable parallel tests: 
build/sbt "set ThisBuild / parallelExecution := true" test
Parallel execution may cause test failures due to port conflicts or shared resources. Use with caution.

​
Code Quality Checks

Delta Sharing enforces code quality through automatic style checking.

​
Scala Style Checks

ScalaStyle checks run automatically with compilation and testing: 
# Style checks run automatically
build/sbt compile  # Runs compileScalastyle
build/sbt test     # Runs testScalastyle
To run style checks manually: 
# Check main sources
build/sbt scalastyle

# Check test sources
build/sbt test:scalastyle

​
Python Code Quality

The Python connector includes linting and formatting tools: 
python/dev/lint-python

​
Test Best Practices

Run Before Commits

Always run tests before committing code changes to catch issues early.

Write Descriptive Tests

Use clear test names that describe what is being tested.

Test Edge Cases

Include tests for error conditions and boundary cases.

Keep Tests Fast

Optimize tests to run quickly for better development experience.

​
Integration Testing

For integration tests requiring a running Delta Sharing Server:
1

Build the server

build/sbt server/compile
2

Configure test server

Create a test configuration file at conf/delta-sharing-server-test.yaml
3

Start test server

build/sbt "server/runMain io.delta.sharing.server.DeltaSharingService -- --config conf/delta-sharing-server-test.yaml"
4

Run integration tests

In another terminal:
python/dev/pytest delta_sharing/tests/test_integration.py

​
Common Test Failures

Solution: Increase JVM heap size
export SBT_OPTS="-Xmx2G"
build/sbt test
Solution: Kill processes using the required ports
# Find process using port 8080
lsof -i :8080

# Kill the process
kill -9 <PID>
Solution: Install package in development mode
cd python/
pip install -e .
Solution: Set up test credentials
export AWS_ACCESS_KEY_ID=test_key
export AWS_SECRET_ACCESS_KEY=test_secret
export AZURE_TEST_ACCOUNT_KEY=test_key
Solution: Fix style violations or update configuration
# View detailed style errors
build/sbt scalastyle

# Auto-format where possible
python/dev/reformat  # For Python

​
Debugging Tests

​
Debug Python Tests

Run pytest with debugger integration: 
# Drop into debugger on failure
python/dev/pytest --pdb

# Set breakpoint in code
import pdb; pdb.set_trace()

​
Debug Scala Tests

Run tests with remote debugging enabled: 
build/sbt -jvm-debug 5005 test
Then connect your IDE debugger to port 5005.

​
Next Steps

Building

Learn how to build Delta Sharing from source

Contributing

Submit your tested changes upstream

CI/CD

Understand the continuous integration pipeline

Build docs developers (and LLMs) love

Get started for freeTalk to us