Data Connectors Overview

​

How Data Connectors Work

• Federate queries across multiple data sources using standard SQL
• Accelerate data by materializing datasets locally with Arrow, DuckDB, SQLite, or PostgreSQL
• Push down operations like filters and projections to source systems for optimal performance
• Connect securely using environment variables, secrets, or configuration parameters

​

Supported Connectors

​

Status Definitions

• Stable: Production-ready with comprehensive testing and documentation
• Release Candidate: Feature-complete with ongoing testing
• Beta: Functional with some limitations; feedback welcome
• Alpha: Early access; expect changes and potential issues
• Roadmap: Planned for future development

​

Configuration Basics

version: v1
kind: Spicepod
name: my-app

datasets:
  - from: postgres:public.users
    name: users
    params:
      pg_host: localhost
      pg_port: 5432
      pg_user: admin
      pg_pass: ${secrets:pg_password}
      pg_db: mydb

​

Common Features

​

Data Acceleration

datasets:
  - from: postgres:public.orders
    name: orders
    params:
      pg_host: localhost
      pg_db: mydb
    acceleration:
      enabled: true
      engine: arrow  # arrow, duckdb, sqlite, postgres, cayenne
      refresh_interval: 10s

​

Query Push-down

-- Spice pushes the WHERE clause to PostgreSQL
SELECT * FROM users WHERE age > 25;

​

Secret Management

params:
  pg_pass: ${secrets:pg_password}  # From secret store
  pg_host: ${env:DB_HOST}          # From environment

​

Performance Considerations

1. Federated Queries: When querying data directly from sources, performance depends on the source system and network latency
2. Acceleration: Enable acceleration for frequently accessed datasets to achieve sub-second query times
3. Push-down: Spice optimizes queries by pushing operations to the source when possible
4. Partitioning: Some connectors support partitioning for parallel data loading

​

Next Steps

• S3 Connector - Query Parquet and CSV files in S3
• PostgreSQL Connector - Connect to PostgreSQL databases
• MySQL Connector - Connect to MySQL databases
• Databricks Connector - Access Databricks Delta Lake and Spark
• Snowflake Connector - Query Snowflake data warehouses
• DuckDB Connector - Query DuckDB databases
• ClickHouse Connector - Connect to ClickHouse
• MongoDB Connector - Query MongoDB collections