Skip to main content
Data sources in Metadb represent external streaming data connections, typically from Kafka topics. These commands allow you to configure and manage how Metadb connects to and consumes data from these sources.

​
Create Data Source

Define a new external data source.

​
Syntax

create data source source_name type source_type
    options ( option 'value' [, ... ] )

​
Parameters

​
source_name
string
required
A unique name for the data source to be created.
​
source_type
string
required
The type of data source. Currently only kafka is supported.
​
options
object
required
Connection settings and configuration options for the data source.

​
Options for Kafka Data Sources

​
brokers
string
required
Kafka bootstrap servers (comma-separated list).
​
topics
string
required
Regular expressions matching Kafka topics to read (comma-separated list).
​
consumer_group
string
required
Kafka consumer group ID.
​
security
string
default:"ssl"
Security protocol: 'ssl' or 'plaintext'. The default is 'ssl'.
​
schema_pass_filter
string
Regular expressions matching schema names to accept (comma-separated list).
​
schema_stop_filter
string
Regular expressions matching schema names to ignore (comma-separated list).
​
table_stop_filter
string
Regular expressions matching table names to ignore (comma-separated list).
​
trim_schema_prefix
string
Prefix to remove from schema names.
​
add_schema_prefix
string
Prefix to add to schema names.
​
map_public_schema
string
Set a new target schema for tables originating in the public schema. For these tables, add_schema_prefix will not be added to the target schema if map_public_schema is set.
​
module
string
Name of pre-defined configuration.

​
Synchronization Process

The new data source starts out in synchronizing mode, which pauses periodic transforms and running external SQL. After no new snapshot records have been received for a significant period of time, the message “source snapshot complete (deadline exceeded)” will be written to the log.
To complete the synchronization:
  1. Stop the Metadb server
  2. Run metadb endsync
  3. Start the Metadb server again

​
Example

Create sensor as a kafka data source: 
create data source sensor type kafka options (
    brokers 'kafka:29092',
    topics '^metadb_sensor_1\.',
    consumer_group 'metadb_sensor_1_1',
    add_schema_prefix 'sensor_',
    table_stop_filter '^testing\.air_temp$,^testing\.air_temp_avg$'
);

​
Alter Data Source

Change the configuration of an existing data source.

​
Syntax

alter data source source_name
    options ( [ add | set | drop ] option ['value'] [, ... ] )

​
Parameters

​
source_name
string
required
The name of an existing data source.
​
options
object
required
Connection settings and configuration options to modify. Use add, set, or drop to specify the operation.
alter data source currently requires restarting the server before it will take effect.

​
Example

Change the consumer group: 
alter data source sensor options (set consumer_group 'metadb_sensor_1');

​
Drop Data Source

Remove a data source configuration.

​
Syntax

drop data source source_name

​
Parameters

​
source_name
string
required
The name of an existing data source.
drop data source currently requires restarting the server before it will take effect.

​
Example

Drop a data source: 
drop data source sensor;

​
List Data Sources

View all configured data sources: 
list data_sources;

​
See Also

Data Mappings

Transform JSON data with data mappings

System Commands

View system status and configuration

Build docs developers (and LLMs) love

Get started for freeTalk to us