Skip to main content

​
Overview

The metadb.log table stores logging information for the entire Metadb system. This table contains all log messages with their timestamps and severity levels, providing a complete audit trail of system activity.
For quick access to recent logs, use the mdblog() function instead. The metadb.log table is useful when you need to query historical logs with complex filtering.

​
Schema

​
log_time
timestamptz(3)
The timestamp when the log entry was written, with millisecond precision
​
error_severity
text
The severity level of the log message. Values include (from least to most severe):
  • INFO - Informational messages about normal operations
  • WARNING - Potential issues that don’t prevent operation
  • ERROR - Errors affecting specific operations
  • FATAL - Critical errors that may cause system shutdown
​
message
text
The complete log message text

​
Usage

​
Query Recent Logs

select * from metadb.log
where log_time > now() - interval '1 hour'
order by log_time desc;

​
Example Output

          log_time          | error_severity |                    message                     
----------------------------+----------------+------------------------------------------------
 2024-03-11 10:45:23.456-05 | INFO           | Stream processor checkpoint completed
 2024-03-11 10:30:15.123-05 | WARNING        | Kafka consumer lag increasing
 2024-03-11 10:15:42.789-05 | ERROR          | Failed to connect to broker kafka:29092
 2024-03-11 10:00:00.000-05 | INFO           | Data source 'sensor' synchronization started
(4 rows)

​
Use Cases

Query all error and fatal messages within a specific time period:
select log_time, error_severity, message
from metadb.log
where log_time between '2024-03-11 00:00:00' and '2024-03-11 23:59:59'
  and error_severity in ('ERROR', 'FATAL')
order by log_time;
Find log entries matching specific keywords or patterns:
select log_time, error_severity, message
from metadb.log
where message ilike '%snapshot%'
  and log_time > now() - interval '7 days'
order by log_time desc;
Get a summary of log entries by severity level:
select 
  error_severity,
  count(*) as total,
  min(log_time) as first_occurrence,
  max(log_time) as last_occurrence
from metadb.log
where log_time > now() - interval '24 hours'
group by error_severity
order by 
  case error_severity
    when 'FATAL' then 1
    when 'ERROR' then 2
    when 'WARNING' then 3
    when 'INFO' then 4
  end;
Track log messages related to a specific data source:
select log_time, error_severity, message
from metadb.log
where message like '%source "folio"%'
  and log_time > now() - interval '1 hour'
order by log_time desc;
Export logs to a file for external analysis:
copy (
  select log_time, error_severity, message
  from metadb.log
  where log_time > now() - interval '7 days'
  order by log_time
) to '/tmp/metadb_logs.csv' with csv header;

​
Severity Levels Explained

​
INFO
Normal operational messages that indicate successful operations or state changes:
  • Stream processor started/stopped
  • Data source synchronization events
  • Checkpoint completions
  • User management actions
​
WARNING
Potential issues that should be monitored but don’t prevent operation:
  • High memory or CPU usage
  • Increasing consumer lag
  • Slow query performance
  • Connection retries
​
ERROR
Errors that affect specific operations but don’t crash the system:
  • Connection failures
  • Query failures
  • Data type conversion errors
  • Invalid configuration
​
FATAL
Critical errors that may cause system shutdown or significant data loss:
  • Database connection loss
  • Unrecoverable stream processing errors
  • Critical configuration failures

​
Indexing and Performance

The metadb.log table can grow large over time. For better query performance: 
-- Always filter by log_time when possible
select * from metadb.log
where log_time > now() - interval '1 day'
  and error_severity = 'ERROR';

-- Use indexed columns in WHERE clause
select * from metadb.log
where log_time between '2024-03-10' and '2024-03-11'
order by log_time;
The metadb.log table is managed automatically by Metadb. Do not insert, update, or delete rows manually. Log rotation and retention policies may be configured at the system level.

​
Comparison with mdblog()

Use mdblog() function for:
  • Quick access to recent logs (last 1-24 hours)
  • Simple time-based filtering
  • Interactive troubleshooting
Use metadb.log table for:
  • Complex queries with multiple conditions
  • Historical log analysis beyond 24 hours
  • Aggregations and statistical analysis
  • Exporting logs to external systems
-- Using mdblog() function (simpler)
select * from mdblog('1 hour');

-- Using metadb.log table (more flexible)
select * from metadb.log
where log_time > now() - interval '1 hour'
  and message like '%kafka%'
order by log_time desc;

mdblog() Function

Quick access to recent log messages

ps() Function

Monitor currently running queries

metadb.table_update

Track table update timing and performance

Monitoring Functions

Query logs and processes programmatically

Build docs developers (and LLMs) love

Get started for freeTalk to us