Skip to main content

​
System Requirements

Before installing the Apache Spark Connector, ensure your environment meets these requirements:
  • Java: 8 or higher
  • Scala: 2.12.x
  • Apache Spark: 3.0 or higher, or Databricks Runtime 9+
If you’re using Databricks Runtime, you can install the connector through the Databricks Libraries interface instead of following the manual installation steps below.

​
Installation Methods

You can install the Delta Sharing Spark connector in two ways:
  1. Interactive Shell: Launch Spark shells with the connector for interactive use
  2. Standalone Project: Add the connector as a dependency in your Maven or SBT project

​
Interactive Shell Setup

The easiest way to get started is to launch Spark with the Delta Sharing connector package.
Launch the PySpark shell with the connector:
pyspark --packages io.delta:delta-sharing-spark_2.12:3.1.0
This automatically downloads and configures the connector for your session.

​
Standalone Project Setup

For production applications, add the Delta Sharing connector as a dependency to your project.
Add this dependency to your pom.xml file:
<dependency>
  <groupId>io.delta</groupId>
  <artifactId>delta-sharing-spark_2.12</artifactId>
  <version>3.1.0</version>
</dependency>
The Delta Sharing Spark connector is compiled with Scala 2.12. Make sure your project uses a compatible Scala version.

​
Version Compatibility

The Delta Sharing Spark connector version 3.1.0 is compatible with:
  • Apache Spark 3.x
  • Databricks Runtime 9.0 and above
  • Scala 2.12.x
Make sure to use the correct Scala version (2.12) when specifying the artifact. Using an incompatible Scala version will cause runtime errors.

​
Profile File Setup

After installation, you’ll need a profile file to connect to a Delta Sharing server. A profile file is a JSON file containing credentials to access shared data. You can obtain a profile file by:
1

Download from data provider

Request a profile file from your data provider. They will provide a .share file containing connection credentials.
2

Try the example server

Download a sample profile file to test with the public Delta Sharing example server:Download open-datasets.share
3

Set up your own server

If running your own Delta Sharing server, create a profile file following the profile file format specification.

​
Verifying Installation

To verify the connector is installed correctly, try loading a shared table: 
# In PySpark shell
table_path = "<profile-file-path>#<share>.<schema>.<table>"
df = spark.read.format("deltaSharing").load(table_path)
df.show()
If you see data output, the installation was successful!

​
Next Steps

Quick Start

Learn the basics of reading shared tables

SQL Usage

Query shared tables using SQL syntax

Build docs developers (and LLMs) love

Get started for freeTalk to us